Inside Arduino - The Build Process

Inside Arduino - The Build Process  Based on Arduino-0022

Level: Intermediate By: Leland Johnson

Introduction

In order to get your program loaded onto your Arduino microcontroller a number of things must occur. First your application must be converted (or built) into a form that can be uploaded to the Arduino hardware. Once built, the code will need to be uploaded into the Arduino's memory. Finally the Arduino should be reset to allow the newly uploaded code to start executing. The entirety of this process involves compilers, linkers, archivers, and uploaders/downloaders. In reality the whole process can get complicated and somewhat confusing.

Within many tools the developer has to create and maintain a makefile. The job of the makefile is to examine the source files that have been changed, then based on that evaluation perform the steps outlined above. For people using the Arduino IDE all of these steps occur with the click of a button - no messy makefiles to create, maintain and execute.

For some of us, understanding what happens during the process that the Arduino IDE performs is important. This blog posting deals with the first part of the process described above - The Build Process. More specifically this blog posting provides some context around what happens when a user selects the "Verify / Compile [Ctrl+R]" option from the "Sketch" menu of the Arduino IDE. Although the upload process is a very important topic, it is beyond the scope of today's posting.

Overview

Before we begin it is important to have an understanding of the Arduino development environment. In general the Arduino version of the Processing IDE does a number of things that makes it easier for individuals new to the world of programming C/C++ to get started. Arduino's IDE allows an individual project, or "sketch" in the Arduino vernacular, to contain multiple files. Each of the files within a sketch is displayed in a separate tab.

Tabs for a file with an extension of .pde have a label that contains the name of the file without its extension. On the other hand tabs for files with any other extension are labeled with the entire file name, including the extension. Why is this import? It presents us with a key understanding...Arduino treats .pde files different from other file types.

During the build process the Arduino IDE perform four major tasks:

1. Combine Files: The .pde files are contatinated together.
2. Transformations: Adds function prototypes and #include statement.
3. Compile Code: Each .c and .cpp is compiled into .o files.
4. Linking: The .o files are linked together with library files.

The result of these tasks being completed is a .hex file that is ready to be uploaded to the Arduino hardware using AVRDUDE - also performed automatically by the Arduino IDE if we choose.

1. Combine Files

The first step in the build process involves concatenating the .pde files into a single program. You should keep in mind that when you build a sketch, the processes are performed against copies of your files that are placed in a temporary directory. On the other hand when you "Upload" a sketch, the build process occurs within the applet directory - a subdirectory of the sketch. The name of the file that is created during the concatenation process is called {sketchName}.cpp; where {sketchName} is the name of your "primary" .pde file.

Reference: Sketch.java

2. Transformations

The Arduino environment makes a few alterations, or transformations, to your code before it is compiled. The only code that is altered during Arduino's preprocessing is the concatinated version of the .pde files ({sketchName}.cpp). This .cpp file is altered in two ways. First off Arduino adds the following line of code before the first line of any of your code, but after any comments:


     #include "WProgram.h"


This line of code imports (includes) a header file that contains some key Arduino definitions and additional includes that make the world of coding for Arduino hardware much easier.

After adding the #include, the Arduino preprocessor creates a function prototype for each functions defined within the {sketchName}.cpp file. These prototypes are inserted after any comments, #include statements or #define statements, but before the first line of code/function.

The final step of the transformation is to append the contents of the main.cxx file for the current target to the bottom of the {sketchName}.cpp file.

Reference: Sketch.java, Preprocessing.java

3. Compile Code

If you are comfortable with GCC, then you will grasp the Arduino compile process quite rapidly. If you are not familiar with GCC this process may take a couple cycles of review, in addition to referring to the WinAVR and GCC reference material (see WinAVR and GCC).

Once the build process completes the Combine Files and Transformations steps, it creates an "includePaths" List that is used for each components it compiles. The include path for the compilation process will consist of the sketch's directory, the target directory ({ArduinoPath}\hardware\arduino\cores\arduino), and the AVR include directory {ArduinoPath}\hardware\tools\avr\avr\include), as well as any libraries ({ArduinoPath}\libraries) referenced in the {sketchName}.cpp file.

Note:  Depending on your installation, the value of {ArduinoPath} will typically be something like "C:\Program Files (x86)\Arduino-0021".

The Arduino IDE individually submits each of the .s,  .c and .cpp files that are part of the sketch, including the {sketchName}.cpp file, to AVR-GCC for compilation. The options used by the Arduino IDE to compile the these .s, .c, and .cpp files are shown below.

.s Files  avr-gcc  -c  -g  -assembler-with-cpp  -mmcu={boardPreferences.get("build.mcu")}  -DF_CPU={boardPreferences.get("build.f_cpu")}  -DARDUINO={Base.REVISION}  -I  -o  .c Files  avr-gcc  -c     -g     -Os     -w     -ffunction-sections     -fdata-sections  -mmcu={boardPreferences.get("build.mcu")}  -DF_CPU={boardPreferences.get("build.f_cpu")}  -DARDUINO={Base.REVISION}  -I  -o  .cpp Files  avr-gcc  -c     -g     -Os     -w     -fno-exceptions     -ffunction-sections     -fdata-sections  -mmcu={boardPreferences.get("build.mcu")}  -DF_CPU={boardPreferences.get("build.f_cpu")}  -DARDUINO={Base.REVISION}  -I  -o

After the sketch has been compiled, the build process will compile each of the imported libraries referenced within the sketch. The build process adds an additional entry to the includePaths List, the utility subdirectory of the library, individually compiles each of the .s, .c, and .cpp files located in the library's directory and utility subdirectory, and then removes the includePaths entry it adds. The object files created during this process are placed in a subdirectory named for the library (i.e. {buildPath}\{libraryName}). The Arduino IDE uses the same compiler options to compile the libraries as it does for the sketch files.

Note:  Some housekeeping needs to occur within the Arduino IDE Java source. The build process adds the utility subdirectory to the includePaths whether the subdirectory exists or not. This is obviously not causing an issue, but is far from being clean. Currently there are only three libraries that have a utility subdirectory; Ethernet, SD, and Wire.

Now that the build process has compiled the sketch and library files, the only remaining component to be compiled is the Arduino core. The core is compiled just like the sketch and libraries; each .s, .c, and .cpp file is compiled individually into .o files. The IDE uses the same compiler options to compile the core as it does for the sketch files. There is one additional step that is performed against the core - the .o files of the core are merged into a single archive (core.a) file using AVR-AR.

Reference: Compile.java

4. Linking

In this context linking is used to describe the process of combining the object and archive files into a single .hex file. In order to accomplish this the Arduino IDE goes through three distinctive stages; ELF File Creation, EEP File Creation and HEX File Creation.

ELF File Creation
The bulk of the linking process occurs during the creation of the .elf file. The Arduino IDE uses AVR-GCC to combine/link the object files and the archive file into a single Executable and Linkable Format output - .elf file.

avr-gcc  -Os  -Wl,--gc-sections  -mmcu={boardPreferences.get("build.mcu")}  -o  -L  -lm

The AVR-GCC options used to perform the linking are show above. You should note that the "-mmcu" option is retrieved from the "boards.txt" file based on the board selected within the IDE.

EEP File Creation
The .eep file is created by performing an AVR-OBJCOPY using the aforementioned .elf file as input. The Arduino IDE executes the AVR-OBJCOPY command using the following program options.

avr-objcopy  -O  -R  ihex  -j  .eeprom  --set-section-flags=.eeprom-alloc,load  --no-change-warnings  --change-section-lma  .eeprom=0

HEX File Creation
The final step the Arduino IDE performs during the build process is to create the .hex file. This is the file that is passed to AVRDUDE when uploading code to the Arduino board.

avr-objcopy  -O  -R  ihex  .eeprom

To create the .hex file the Arduino IDE executes the AVR-OBJCOPY utility with the options listed above. The .elf is used as the source of information to create the .hex file.

Reference: Compile.java

Conclusion

The Arduino Team did an excellent job creating a user friendly experience. A big part of that is removing the burden of creating and maintaining makefiles for your embedded systems work. The Arduino IDE is well suited for the novice software develop and of course the hardware platform is a great tool for many fun and exciting hobbyist endeavors. From a professional perspective the tool does have its short-comings. Overall I enjoy Arduino, and the team that developed it.

---

The information from this blog was derived from a number of sources. The primary source of information was the actual Arduino IDE source code - specifically Arduino-0022. In each of the above sections. The specific source files that where used to obtain the information are listed. I hope that you find this information in a format that is understandable, beneficial and relevant. If you enjoyed this post please let me know. I would also appreciate your feedback on the content, suggestions for improvement, and adjustment for clarity or accuracy.

Inside Arduino - The Build Process  Based on Arduino-0022

Level: Intermediate By: Leland Johnson