Improving Linux Scripting

I was asked how I have been able to improve my Linux scripting abilities for use with Linux and figured I would turn it into my first tech-focused blog post.

To start off for any Linux scripting, it’s important to know your way around Linux command line so that when it comes time for your code to execute system commands, you know that it’s going to do what you want it to do. This also includes knowing your way around the file system, a bit on interacting with running processes, as well as piping and stdin/stdout/stderr redirection. Distribution-specific documentation and wikis are very useful, but I also like more broad-reaching Linux resources that aren’t tied to a specific distribution: LinuxQuestions.org, Unix/Linux StackExchange, and various G+ groups. One website I have consistently used as a reference for redirection has been: catonmat.net – Bash One Liners Explained (Part III) since it gives nice diagrams detailing what exactly is going on. And within Linux, you always have ‘man’ pages to fall back on.

For scripting, I recommend starting out with focusing solely on Bash even if you wish to branch out to other scripting languages (Ruby, Python, Perl, etc.). Scripting in Bash will keep the focus on what you are coding, rather than all of the intricacies of whatever language you’re using; white space, punctuation, and any other language-specific particulars. Bash will give you a strong foundation on which to expand out to the scripting language of your preference, and will improve your Linux knowledge and efficiency at the same time.

I recommend setting up a server of your own which you can use as a sandbox; this can be actual hardware, or something virtualized on your dev system or in the cloud. As an example, my website here runs on my own hardware running Gentoo Linux, with a ZFS filesystem (for snapshots and, in my opinion,] easy partitioning, quota management and NFS sharing management). It also runs Xen hypervisor under which this website is running on Nginx, PHP/WordPress, and MySQL (a separate Xen guest). All of this has given me a perfect environment to not only learn, but with plenty of things that can benefit from scripting and automation. Backups were the first process I focused heavily on scripting and I still use those original Perl scripts I wrote to do it: Perl: Backup Scripts. Backups are a good place to start because everyone should have backups, and you have a vested interest in ensuring you have good functional backups.

Aside from larger scale routine tasks, you may begin to accumulate a bunch of smaller repetitive tasks that get… well… repetitive, and you’ll want to start automating those tasks. As an example, I compile all of my own kernels and, while the actual configuration of the kernel is a manual process of selecting the necessary modules, the compiling, upgrade, and installation process can be automated. For most upgrades, it’s a matter of me downloading the latest kernel version, extracting it, copying over the previous working config, compiling, and installing the new kernel. You can see an example of that here with my Bash script which handles the upgrade process for my primary server and Xen guests: Bash: Kernel_Install_dAnarchy. All that really needs to be added is an automatic download and extraction of the new version, and this script could easily be scheduled with a cron job to keep my server’s kernel up to date.

The above script gives some good examples of variable use, a conditional (L19-22), and a loop (L45-48), and runs through the process of setting the kernel version, compiling it, and installing it into the appropriate location. It then runs through the configuration upgrade for each of my Xen hypervisor guests. The next reboot will bring everything up in the new version.

There are many guides online to assist with general Bash scripting knowledge, and that’s far too much information to go through here. In general, though, I mostly learn as I go based on whatever it is I need to accomplish. As I come to problems that I don’t know how to solve, I find ways other people have already done similar actions and begin experimenting with what I find.

This doesn’t mean just copy/pasting from the internet; not only may you end up running something you don’t quite yet understand, but you won’t be learning anything new. Experimenting with what I find in an online search not only yields multiple ways of doing something, but those experiments stick around on my hard drive for future use.

I have taken a few online courses through Udemy and Linux Academy (I went to university for other subjects), and while those are solid options that have taught me some good basics, most of what I have learned has been organically absorbed by doing it.

As for advancing on to scripting languages beyond Bash, I moved on to teaching myself Perl since my current place of employment is a Perl dominant environment. At work, I began by finding a few small manageable bug fixes that required only 1-5 lines of code change. That sounds pretty meaningless, but it did a few things to get me started. Firstly, it forced me to not only read other people’s code (which is often not consistent even within single company), but also to understand what the entire script, or at least the immediate method is doing. It isn’t about just changing a few lines, but more about understanding the language and the overall task the script is performing.

At the same time, I began upgrading some of my Bash scripts into Perl, giving me experience of writing scripts from the ground up. The backup scripts I referenced above were originally some smaller Bash scripts just handling website and MySQL backups. I used these as a means to skill up in Perl and also get some additional experience on automating those backups within virtual guests.

And my Ruby experience came about because I wanted to learn a language that was much simpler for me to understand and write. I also came up with the idea to automate OpenStack management, and that gave me a solid project on which to learn Ruby: that turned into dAnarchy_sys. This happened around my birthday last year when I signed up for Ruby and a Ruby on Rails Udemy courses, so that is about the amount of time it has taken me to get where I am at with Ruby.

I have also spent some time doing CodeFights and CodeWars which give some good challenges both on your language of choice, and on the logic involved with solving the problem. I personally don’t do them as a competitive/score based thing, though. If I run into a problem I can’t logically solve, I search and find a solution. The point in me spending my time doing those is that I learn something new more than testing whether I already know how to do something.

Overall, no matter what scripting language you wish to learn, or what you want to do with it, I think that having some actual project ideas gives a good foundation on which to learn. And it also gives you a good historical record of your progress to gauge how much you have improved, or how much further you could improve.

Leave a Reply

Your email address will not be published. Required fields are marked *