Been building a new mirror (internal) host @ work, and its been an adventure. This isn’t a public mirror, and we’re not interested in mirroring all releases of Distro X, but rather only a handful or one. We basically want to mirror:

• CentOS 5 & 6
• EPEL 5 & 6
• PuppetLabs Repos for el5 & 6
• Ubuntu LTS Release(s)

Now, the mirror box is a Centos6 host, so this is pretty simple when it comes to the CentOS mirrors, thats comeing in another post. We do some funky things like want a fully sync’d mirror, but also want point-in-time snapshots so we have a static source to build compute nodes from (while storeage/webserver get the up-to-date repos).

In anycaase, building a Ubuntu mirror, other than just rsync’ing archive.ubuntu.com/ubuntu, on a non-Ubuntu/Debian host is non-trival it seems, just like using something other than rsync to mirror yum repos on something w/o yum installed would be awkward. However, I’ve found a solution:

debmirror

DebMirros is really just a Perl script, and I’m wrapped it up in script for our setup. Took a little bit to get going, though:

yum install perl-libwww perl-Compress-Zlib perl-Digest-SHA1 perl-Net* rsync perl-LockFile-Simple perl-Digest-MD5-M4p
wget http://archive.ubuntu.com/ubuntu/pool/universe/d/debmirror/debmirror_2.10ubuntu1.tar.gz
tar -xzvf debmirror_2.10ubuntu1.tar.gz
...
cd debmirror-2.10ubuntu1
make
cp debmirror /usr/local/bin/
cp debmirror.1 /usr/share/man/man1/
cpan install Net::INET6Glue (couldn't find this in yum). 

Okay, debmirror now works, but needs that wrapper script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/bash
arch=amd64
section=main,restricted,universe,multiverse
release=lucid
server=us.archive.ubuntu.com
inPath=/ubuntu
proto=http
proxy=http://proxy.local:8888
outpath=/var/www/repos/ubuntu

debmirror       -a $arch \
                --no-source \
                -s $section \
                -h $server \
                -d $release \
                -r $inPath \
                --progress \
                --ignore-release-gpg \
                --no-check-gpg \
                --proxy=$proxy \
                -e $proto \
                $outPath

Make that executable, run it, and we’re off! First sync is running now. Next up is to see how quickly a resync happens, and schedule it to run daily. Hello internal Ubuntu Mirror!

The really nice thing is when the next LTS comes out (Precise Pangolin), I can simple add it to the list of release like

release=lucid,precise

And we’ll start mirroring that!

Next up, convuluted ways of builing many many Centos Mirrors on one box for various reasons

While I’m yet to find the holy grail of getting a KVM host to be aware of which ‘dom0’ its being hosted on, this bit of ruby for a custom puppet facts is a step there, the other way around:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

require 'facter'
begin
        Facter.compute_node
rescue
        Facter.loadfacts()
end


unless  `rpm -qa | grep ruby-libvirt`.empty?
        if (Facter.value("puppet_class_kvm_host") == "true") then
                Facter.add("kvm_vms") do
                        setcode do
                                require 'libvirt'
                                conn = Libvirt::open('qemu:///system')
                                if (conn.num_of_domains == 0 ) then
                                        kvm_vms = "NO VMS"
                                        conn.close
                                        kvm_vms
                                else
                                        vm_doms = Array.new
                                        conn.list_domains.each do |domid|
                                                dom = conn.lookup_domain_by_id(domid)
                                                vm_doms.push(dom.name)
                                        end
                                        vm_doms=vm_doms.join(" ")
                                        conn.close
                                        vm_doms
                                end
                        end
                end
        end
end

Which will return a nice fact like:

kvm_vms => Host1 Host2 Host 3

or if none are running:

kvm_vms => NO VMS

its ugly, but it works!

One question we get asked at wrk a good bit centers around finding compute resources that match a specific set of specifications.Since the compute system we haev are of all sort of various ages and hardware types, jobs that need 8GB of memory might not work everywhere, but most places.Same goes for a specific number of cores (or, minimal #), per node. Some have 4, some have 12+.

Well, facter, and therefore puppet, knows about these things. And there is a REST api we can query. And I needed an excuse to do a little ruby:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

#!/usr/bin/env ruby

require 'yaml'
require 'puppet'

puts "Welcome to the Compute Finder script!"
puts "This script aims to help you locate compute nodes based on simple requirements"
puts "Such as the minimum amount of RAM or # of processor cores"
puts "---"
puts "---"
puts "What minimum amount of ram would you like?"
puts "(in GB, leave blank for no minimum)"
mem=gets.chomp
puts "What minimum number of processor cores would you like?"
puts "(leave blank for no minimum)"
procs=gets.chomp

sleep 1

search ="facts.compute_node=true"
if mem != ""
  search << "&facts.memorysize.ge=#{mem}"
end

if procs !=""
  search << "&facts.processorcount.ge=#{procs}"
end

if (procs == "" and mem == "" )
  puts "You must specify some requirments!"
  exit
end
puppet_base = "https://puppet:8140"
path="/production/facts_search/search?"


puts "Finding matching nodes, please wait..."
cmd = "curl -s -k -H \"Accept: yaml\" \"#{puppet_base}#{path}#{search}\""
ans = %x[#{cmd}]
nodes = YAML::load(ans)
nodearr = []
nodes.sort.each do |node|
  nodearr.push(node)
end
puts nodearr
puts "Would you like more info on the nodes found? This may take bit longer..."
puts "(y/n)"
more=gets.chomp
if more == "y"

  node_array = []
  nodes.each do |node|
      node_url="https://puppet:8140/production/facts/#{node}"
      node_cmd="curl -s -k -H \"Accept: yaml\" #{node_url}"
      node_ans= %x[#{node_cmd}]
      node_array << YAML::load(node_ans)
  end

  results =[]
  node_array.each do |node|
      name = node.name
      facts = node.values
      ram=facts['memorysize']
      procs=facts['processorcount']
      results.push("Host: #{name} RAM: #{ram} ProcessorCores: #{procs}")
  end
  results.sort!
  puts results
end

The output of which looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

Welcome to the Compute Finder script!
This script aims to help you locate compute nodes based on simple requirements
Such as the minimum amount of RAM or # of processor cores
---
---
What minimum amount of ram would you like?
(in GB, leave blank for no minimum)
64
What minimum number of processor cores would you like?
(leave blank for no minimum)
12
Finding matching nodes, please wait...
node01.local
node02.local
node03.local
node04.local
Would you like more info on the nodes found? This may take bit longer...
(y/n)
y
Host: node01.local RAM: 94.28 GB ProcessorCores: 12
Host: node02.local RAM: 94.28 GB ProcessorCores: 12
Host: node03.local RAM: 86.39 GB ProcessorCores: 12
Host: node04.local RAM: 251.89 GB ProcessorCores: 32

So, obviously,we use puppet a great deal. One thing a few of us have wanted for a while was the ability to ssee that nodes/node groups would be affected by a change to a particular class. In other words, what hosts get $This class?

Now, nodes are aware what classes they get, but the puppet master seemingly is not, not really. It compiles teh catalog on a per-run basis, and while durring that it DOES know what classes a system gets, it doesn’t really store that info. So, a simple custom fact to pull that data in from each host:

1
2
3
4
5
6
7
8
9
10

#list puppet classes, on per fact
require 'facter'

IO.popen('cat /var/lib/puppet/classes.txt | tail -n+3').readlines.each do |line|
        Facter.add("puppet_class_#{line}") do
                setcode do
                        "true"
                end
        end
end

So, now we get, for each host, a bunch of facts like:

1

puppet_class_syslog => true 

Reported from facter -p, and therefor sent back to teh puppet master, into a DB, which we can query directly, via teh puppet api, or via Foreman.

So, this isn’t “new”, I wrote it a few weeks back, but, well, meh, figured I’d share. First “useful” ruby-instead-of-bash script for a little automation/keeping things consistent.

@ work we have the need to move a whole mess of storage (just under 100TB) from one thing to another, and to preserve quotas as we do so. Most of the quotas are simple 1TB tree/project quotas. Some are bigger, but by default we want to start @ 1TB and then we can bump the ones needed. Anyways, I need to do this for about 8 groups of 5-20 “projects/trees”, so by hand isn’t really an option. Hence the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#Usage: mkscartch <project/group name> 
#Makes the needed project/projid entires turns on the project quota, and sets a 1TB default hard/soft limit. 
if ARGV.length != 2
  puts "usage: mkscartch <project/group name> <full path to directory>"
  puts "This script takes a lab/group name, and a path as arguments"
  puts "It will make a directory for the lab/group at the path, and turn it into a default 1TB project quota"
  exit
end

new_scratch = ARGV[0];
new_path = ARGV[1];
project_nums=Array.new
#first, we need to figure out what project ID to start with
File.open("/etc/projects").reverse_each { |projects|
        project_nums.push(projects.split(':')[0])
}
last_project = project_nums[0].to_i;
new_project = last_project+1
project_line="#{new_project}:#{new_path}"
projid_line="#{new_scratch}:#{new_project}"
mkdir_cmd="mkdir #{new_path}"
quota_set="xfs_quota -x -c 'project -s #{new_scratch}'"
quota_limit="xfs_quota -x -c 'limit -p bhard=1024g bsoft=1024g #{new_scratch}'"

sleep(1)
puts "Adding #{project_line} to /etc/projects..."
File.open("/etc/projects", "a") do |f|
        f.puts(project_line)
end
sleep(1)
puts "Adding #{projid_line} to /etc/projid..."
File.open("/etc/projid", "a") do |f2|
        f2.puts(projid_line)
end

sleep(1)
puts "Making #{new_path} with #{mkdir_cmd}"
Dir::mkdir(new_path)

sleep(1)
puts "Setting quota with #{quota_set}"
system "#{quota_set}"

sleep(1)
puts "Setting quota limits @ 1TB with #{quota_limit}"
system "#{quota_limit}"

So, I can, from the place I want to sync from, do something like:

1

$ for i in *; do /usr/bin/mkscratch.rb $i /new/location/$i; done

And be done with it. Really basic, I know. But, first thing I would have normally shell scripted done in ruby instead. And you know what? I liked it, Its nice. So there.

Weekend time and time to do a bit of house clenaing. For a bit the macbook pro has beend draining a bit too fast and getting a bit too hot. I’m not convinced its hardware. I did an upgrade from Snow Leopard to Lion, and there is a good bit of cruft around. So, I decide to do a nice clean wipe, fresh Lion install, and keep it basic.

How basic?

I normally tend to say/think that all I really need to do 98% of my day to day work is Chrome and a Terminal. One of those is built in. So, a clean reinstall, install Chrome (which I’ve rebased to have very few extensions etc), install RVM, get my gems, pull dotfiles/etc from github, install homebrew for building UNIX app (Thanks to Phil for the recommendation, I already like it far better than macports), get XCode from the app store, and well, thats about it. Really.

I’m going for the minimal number of apps installed. No Office if I can get away with it (Google Docs), no Mail.app/Thunderbird (Gmail/Google Apps), No GUI text editors (VIM!). I’m goign to try to avoid Dropbox, though that won’t last long, but I’m going to give it a thorough cleaning and move lots to github/a private git server.

Thats the next task up, a git server/repo @ home. Might got KVM on my home server, and stand up a handful of minimal VMS, for testing, playing around, sperating services a bit.

More to come!

Made a little progress:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

nichols2::LSCI-NICHOLS2-2 { 17:00:53 Fri Sep 30 }
~/Development/rudo-> rake init
RuDo init started, creatding SQLite DB...
nichols2::LSCI-NICHOLS2-2 { 17:00:56 Fri Sep 30 }
~/Development/rudo-> rake new_list["Todo"]
New List "Todo" created with ID#[1]
nichols2::LSCI-NICHOLS2-2 { 17:01:05 Fri Sep 30 }
~/Development/rudo-> rake new_list["Work"]
New List "Work" created with ID#[2]
nichols2::LSCI-NICHOLS2-2 { 17:01:09 Fri Sep 30 }
~/Development/rudo-> rake new_list["Home"]
New List "Home" created with ID#[3]
nichols2::LSCI-NICHOLS2-2 { 17:01:17 Fri Sep 30 }
~/Development/rudo-> rake lists
RuDo has the following lists:
1
Todo
2
Work
3
Home

It works. Sending the push to github for the day.

Puppet is all ruby base. This blog (octopress), is all ruby based. Ruby seems neat, and I’d like to learn a new language, and I know the only way I’ll evern learn a new one is to have some goal to accomplish.

So, since I’m always looking for a new Todo/Task list option, I’m thinking of writing my own….

I liked todo.txt, and honestly may go back to that at some point, but first I want to try this out. I figure a task list is simple, and I only have the following requirments:

• Multiple tasks lists/”tags” for tasks, as I want to seperate work from personal, etc etc
• Simple done/not done functionality.
• no need for due dates or priority
• store teh date created and date marked done, incase i want them later.

So, For now I’m thinking:

• Write this in Ruby.
• Sqlite .db file to store the data
• use a Rakefile to run everything?, ala “rake new_list[“work”], rake new_task[“get back to work”,”work”], rake complete[task#] ” ?

So, for database tables, I’m thinking:

• Tasks table store, well, tasks (taskid, status, title, notes, listid(or listids?), date created, date completed)
• lists table has task list names matched to listids.

Thats it!

ruby + sqlite should to the trick. I can put this in dropbox and keep it in sync everywhere. This doesn’t solve the mobile device use case, but, thisis for learning more than anything.

Okay, so, on another system, I did (roughly):

1
2
3
4
5

git clone -b source <remote-url> octopress
  cd octopress
  <run through the ocotpress getting started guide if neede dto get all teh ruby bits installed,no need to rake install>
  rake new_post["title"]
  rake setup_github_pages

and here it is.. Neato. And The publishing routine is:

1
2
3
4
5

rake generate
  rake deploy 
  git add /path/to/new/stuff
  git commit -a 
  git push origin source

I think thats it…

trying out octopress a bit. mostly to force myself to learn git way better. lets see…