What I Wish I’d Known When I Was an Embedded Linux Newbie

Here are some tips compiled from our seasoned engineers on what they wish they’d known about embedded linux back when they were “newbs.” Newcomers and seasoned veterans alike should get some good nuggets of information and possibly a fun perspective looking back at our own humble beginnings. We’ll try not to overwhelm you as we make our way through the list. We’re not here to rewrite the books, but do want to provide a personal perspective. If you’re in the camp of people who’ve been using desktop Linux, just be aware embedded Linux is a different animal, especially when it comes to space constraints, different CPU architecture (ARM), resilience to sudden power outages, and inability to install any mainline Linux kernel or distribution you please. And maybe you’re in the microprocessor camp moving towards a more generalized and capable embedded Linux system. Either way, we’ll assume you have at least some knowledge of Linux as we walk through this guide.

Getting Around
- Serial Console (RS-232)
  - Understanding UARTS like RS-232, RS-485
- Secure Shell
  - SSH
  - SCP
- Screen and Tmux
Editing Files
- Vi and Vim
- Emacs
Disk and File Corruption
NFS
Programming
- ctags and cscope
Kernel Compiling
NAND Flash Considerations
Checking Space
Reading/Writing Images
Distributions and Tarballs
Recommended Reading

Getting Around

One of the first things you’ve probably already come to notice in an embedded Linux product is the lack of a nice desktop environment. No GUI here, my friend! Here, the command line is king. Just as you’re probably very proficient with a mouse, you’ll come to develop muscle memory to make you as proficient if not more using a keyboard. Right now, you’re probably feeling more like this finger typer here, and that’s okay! We’re here to tell you things are going to get better.

Serial Console (RS-232)

One of the first things you’ll use is a serial console. Yes, ‘serial console’ as in the archaic Microsoft Hyperterminal program to connect to your board to get a command prompt, or shell in Linux-speak. The serial console is the most trusted method of communicating with the board since it’s not prone to network failure as is SSH and Telnet, plus you’ll be able to see both startup and shutdown messages. For connecting via serial console, take a look at minicom (Linux/Mac/Cygwin) or PuTTY (Windows). A serial console must be configured to talk at the same rate of speed as the board you’re connecting to. It seems the most common is 115200 Bps with no parity bit and no hardware or software flow control. This is what minicom setup (minicom -s) looks like on a Mac:

You’ll need some parts for this: A null modem cable and a serial port on your computer. If you don’t have one, you can get a USB to Serial convertor.

Pro tip: resize(if available) is amazing for serial consoles that don’t auto-grab the terminal size. Home

Understanding UARTS like RS-232, RS-485

It’s advisable to brush up on your UART communication skills, since connecting to RS-232 serial console is much the same as connecting to a standard TTL-level UART or RS-485. Some resources which may help your understanding:

Secure Shell

A heavily accessed tool in the toolbelt is secure shell and you’ll want to be sure to become efficient with it. It comes down to two main commands which use the same underlying technology: SSH and SCP.

SSH

SSH allows you to connect to your board’s shell prompt using the network. It uses port 22 by default, and you can setup port forwarding in your router to allow for remote access to the board. Typical usage is very simple:

ssh <user>@<ip or hostname>
ssh root@192.168.1.50

SCP

SCP can be thought of as “Secure Copy.” This allows you to transfer files to and from your board. If you can SSH to your board, you can copy files. Typical usage:

scp <filename> <user>@<ip or hostname>:<directory>
scp foobar.txt root@192.168.1.50:/root/

Screen and Tmux

Instead of having multiple windows open to the same board, try using screen or tmux. Each of these programs can provide multiple shells for a single connection with the added bonus of keeping your shells active and open even when your connection is closed. Screen usually comes preloaded and is smaller than Tmux, and is usually sufficient. Tmux is more powerful and has it’s own advantages. Since screen is so readily available, it’s nice to dedicate muscle memory to.

Editing Files

Since you’ll be spending a lot of time in the command line, you’ll want to try and become proficient at using a non-GUI editor. There are two main contenders here: VIM and Emacs. Do a Google search for VIM vs Emacs and you’ll quickly find there are two separate camps each very passionate about their preferred editor. The engineers here at Technologic Systems are mostly in the VIM camp, but encourage you to decide for yourself which is right for you.

Vi and Vim

Most embedded Linux boards will come preloaded with at least ‘vi’, the original ‘vim’. Vim is simply ‘vi improved.’ A vim install takes a bit more storage, but in this author’s opinion it’s totally worth every extra byte.

Derek Wyatt has a fantastic and entertaining set of videos for learning Vim, from basics to advanced usage.

http://derekwyatt.org/vim/tutorials/

Emacs

Emacs has a more traditional word editor feel than Vim which many beginners find comforting. If it’s not installed by default, it easily can be as it’s in most package management systems like apt-get and yum.

There’s a guided tour of Emacs over at GNU.org if you’re interested in learning more.

Disk and File Corruption

You’ve seen this message before. Admit it, you’ve unplugged a disk without ejecting it first because you think that message is for chumps and you’ve never had any problems with data on your USB thumb drive. In the embedded Linux world, this is an absolute must, as disks are prone to data and filesystem corruption when the carpet is pulled from underneath. This includes sudden power outages. You can attempt to use fsck to repair the disk, which works for a while, but you’ll inevitably find yourself in a situation where it cannot repair the disk any longer. We’ve written a whitepaper on filesystem corruption and ways to prevent it from happening.

A utility to get in the habit of using is sync, which commits buffer cache to disk. You can read more about it in the sync man page. Home

NFS

Standing for Network File System, NFS is a great tool to use for accessing files between two computers on a network. The definition is simple, but the implications are many. For example, you can setup (or export) an NFS share containing a tarball of a Linux distribution and setup your board to load it. You can also setup an NFS share to edit your project files in an IDE on your desktop machine and then compile it on the embedded board without the need for manual copy/paste. The Ubuntu community has a really nice article, “Setting Up NFS How To.”

Programming

Whoah. What an overwhelming topic. It’s an exciting one, and certainly important, but there are so many different languages, platforms, memory addresses, processors, styles, etc., it’d be very difficult to not overwhelm a reader in this article. Let’s just say that in the embedded Linux world, the language of choice is usually C because of low-level hardware optimization. In fact, the Linux kernel is written in C, a decision the creator Linus Torvalds has a very non-apologetic stance on. Because of limited resources, efficient and fast (aka inexpensive) code is sought after. For example, floating point arithmetic is generally frowned upon because it involves a lot of layers, including the kernel, to make floating point happen which is very expensive and time consuming. If your CPU has a hardware floating point unit, then it’s not really a problem.

One pitfall that a newbie might fall into is NIH (not invented here) syndrome, where they feel they have to reinvent the wheel. In actuality, there are a ton of existing resources and existing libraries (ie. key/value storage like Berkely DB, send/receive data via HTTP or FTP). A new development project ought to be preceded by an extensive search or research phase before jumping in. Some Linux API calls exists that a lot of people don’t realize, like sched_setscheduler(), mlock(), mlockall(), etc. Also, having an understanding about /dev/mem makes development in userspace much easier and foregoes the need to develop kernel drivers. Resist the temptation to put more and more of your embedded application into the kernel and stick with the bare minimum and you’ll almost always be better off in the long term. Along those lines, don’t think of your device as a monolithic application and instead separate things into processes. Not everything needs to be optimized, so don’t be afraid to use tmpfs filesystem or pipes to communicate or shell scripts. Really, it comes down to not underestimating the virtue of simplicity. Use layers, but use them strategically, being sure not to add them until necessary else your system will almost always get slower, have difficulty troubleshooting failure modes, and result in higher demands on hardware. Layering is an underutilized advantage of embedded programmers coming from microcontrollers and over utilized by developers coming from a PC or sysadmin background. Home

ctags and cscope

If you’re doing anything related to software development, be sure to take a the time to learn about ctags and cscope. These give you a method for jumping to and from function and variable declarations across your project files and can work together. Both VIM and Emacs work with ctags (etags for Emacs). You can learn more about ctags and cscope by reading our guide, Tag Jumping with ctags and cscope.

Kernel Compiling

Eventually in your embedded Linux travels, you’ll come across some instructions that say you need a particular kernel module. When you find out your kernel doesn’t have it, it’s intimidating to think of what you’ll need to do in order to compile it yourself from scratch. It’s not actually as scary as you might think, and it’s a very good way to improve your understanding of Linux. Each single board computer or System on Module will have slightly different instructions and different versions of the kernel, and Technologic Systems’ engineers have worked hard to write guides on several of their products. The guides aren’t incredibly long, and go over things like choosing the module to compile via make menuconfig and inserting the module via insmod when you’ve successfully compiled them. Take a look at the following guides as a starting point, and then reach out to a support channel to find out more specific information about your board.

NAND Flash Considerations

NAND flash is really fast, but don’t write to it a lot as it’s not as fantastic as it seems. eMMC flash is more reliable, doubly so when configured in SLC mode.

Checking Space

Disk space is usually constrained in an embedded Linux device. You may have the ability to increase storage using an SD card or USB thumb drive, but likely, you’ll be booting from an on-board flash drive which might only have 4 GB or less. Make sure you’re aware of file sizes and their units, for example MB is different than MiB (MiB = Mebibyte = 1024 KiB; MB = MB = Megabyte = 1,000 KB). Commands like ‘du’ and ‘df’ help identify how the storage is being utilized. You can also remove and purge unnecessary locales, or language translations to free up space. If you’re in a Debian-like environment, be sure to use tools like apt-get clean and autoremove to keep things on the up and up.

Reading/Writing Images

Undoubtedly, you’ll need to restore a factory image or load a prepared image onto your embedded Linux device. Usually the preferred tool for generating these images is dd. We have a guide that will walk you through how to do this in various platforms. See How to Write an SD Card Image (Linux, Windows, Mac OSX).

Distributions and Tarballs

Since Linux is a file based OS, you can create a tarball of your distribution and extract it to either a prepared partition or NFS filesystem and it’ll boot. You can create a tarball using a command similar to:

tar -cvpf /root.tar --directory=/ --exclude=proc --exclude=sys --exclude=dev/pts --exclude=backups .

Conclusion

Hopefully you feel like you’ve come away with some great tips to get you going in the embedded Linux world. There’s still so much to learn, so get out there and start exploring! Home