***************
The Scotty tool
***************
Scotty
======
``SimpleCore open technology tool for you`` or short ``Scotty`` is a tool used to
operate with the SimpleCore BSP, helping to download sources and build BSP images. It
relies on docker to provide a reproducible build environment.
``Scotty`` can either be used as a single command to build a full image or as a
helper to setup Yocto sources and open a shell with a Yocto environment.
Getting started
---------------
The scotty tool relies on a docker container to embed as much as possible the
software dependencies it relies on. As a consequence, you will need to make sure
to install `Docker<https://www.docker.com/>` on your system and make sure your
user has rights to use docker.
The scotty tool also needs `OpenSSL<https://www.openssl.org/>` to function properly.
On Ubuntu 20.04 platforms, they can be installed with following commands::
$ sudo apt install docker.io openssl
$ sudo gpasswd -a ${USER} docker
In addition, if you plan to use QEMU images, it is advised to make sure the
current user is able to use `KVM <https://www.linux-kvm.org/page/Main_Page>`_,
having write permissions to ``/dev/kvm``. On Ubuntu 20.04, this can be done by
adding your user to the ``kvm`` group, with the following command::
$ sudo gpasswd -a ${USER} kvm
.. note::
After adding your user to the docker or kvm group with the ``gpasswd``
command, you will need to open a new session to apply your new privileges.
Installing the scotty tool can be easily done with ``pip``
.. code:: shell
pip install --user avnet-scotty
upgrading an existing installation can be done with
.. code:: shell
pip install --user --upgrade avnet-scotty
Alternatively, the current development version can be obtained by cloning
`<https://github.com/avnet-embedded/simplecore-tools>`_.
.. note:: Remove the following paragraph once the BSP is open-sourced.
To allow ``scotty`` to download the associated docker image, you will need to run
a docker login command, as documented on
`<https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry>`_
Scotty subcommands
------------------
Several commands are supported by Scotty, with different goals:
- ``update``: download or update the Yocto sources. Will be triggered
implicitly with default parameters if no sources are found.
- ``setup``: Setup the build directory. Will be triggered implicitly with
default parameters if the build directory has not been setup.
- ``bitbake``: Build a Yocto target.
- ``shell``: Run a shell in the Yocto build environment.
- ``command``: Run a command in the Yocto build environment.
- ``docker-update``: Update Scotty build container.
- ``info``: Print out information.
Setup your build
----------------
With
.. code-block:: bash
$ scotty setup
you can configure your build through our interactive UI.
.. note::
.. code-block:: bash
$ scotty setup --force
Will allow you to redo the configuration at any point
Building a full image
---------------------
A single command is enough to download the sources and build an image:
.. code-block:: bash
$ scotty bitbake core-image-minimal
Built images can then be found in ``build/build/tmp/deploy/images/``.
The build target can be customized:
.. code-block:: bash
$ scotty bitbake core-image-minimal
$ scotty bitbake -- core-image-minimal -c populate_sdk
.. note::
The available images/SDKs can be displayed by running
.. code-block:: bash
$ scotty info images
Recommended hardware setup
^^^^^^^^^^^^^^^^^^^^^^^^^^
For building software pacakges with ``Scotty`` we recommend the following minimal hardware setup
For base images
- 4 Cores / 8 threads CPU
- 16GB RAM
- 200GB HDD
For SDKs
- 16 Cores / 32 threads CPU
- 64Gb RAM
- 500GB HDD
.. note::
Other combinations do work as well, but keep in mind that we at least require 2GB of RAM per available CPU thread
Using Scotty as a helper
------------------------
``Scotty`` can be used to open a shell with a sourced Yocto environment:
.. code-block:: bash
$ scotty shell
It can also be used to run a single command without opening a shell:
.. code-block:: bash
# This is equivalent to scotty bitbake core-image-minimal
$ scotty command bitbake core-image-minimal
$ scotty command bitbake-layers show-appends
Updating sources
----------------
Scotty will not modify downloaded sources by itself after the initial setup.
Building an image with up-to-date sources can be done using following commands:
.. code-block:: bash
$ scotty update
$ scotty bitbake core-image-minimal
Advanced configuration
----------------------
Scotty can be used to download extra sources, add additional layers or tweak the
configuration.
Supported update arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^
- ``--manifest-branch``: Set repo branch used for repo init.
- ``--manifest-url``: Set repo url used for repo init.
- ``--manifest-name``: Set repo manifest used for repo init.
- ``--extra-download``: Add an extra git to download. If this is a layer, it can
then be used in setup with ``--extra-layer``.
Supported setup arguments
^^^^^^^^^^^^^^^^^^^^^^^^^
- ``--build-dir``: Set build subdirectory, subdirectory of ``build``. Can be
used to have different builds in the same ``build`` folder.
- ``--machine-dir``: specify the directories where scotty should look for
supported machines.
- ``--extra-layer``: Add an extra local layer.
- ``--extra-conf``: Add an extra configuration entry in local.conf.
- ``--extra-env``: Pass on additional environment variables.
- ``--sstate-mirrors``: Do use any sstate mirror (default = false).
- ``--features-layers-set``: The set of Avnet Embedded extra layers to use.
- ``--machine-dir``: specify the directories where scotty should look for
supported machines.
.. note::
The currently available ``--features-layers-set`` can be displayed by running
.. code-block:: bash
$ scotty info feature-sets
Supported bitbake, shell and command arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- ``--build-dir``: Set build subdirectory, subdirectory of ``build``. Can be
used to have different builds in the same ``build`` folder.
Scripting configuration
-----------------------
It might be useful to skip the configuration menu, so Scotty can be used in a
scripted way. To help with this, most configuration can be set either through
arguments or environment variables.
Supported arguments
^^^^^^^^^^^^^^^^^^^
The ``--features-layers-set`` of the ``setup`` subcommand can be used to bypass
layer sets selection.
Supported environment variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- ``DISTRO``: Can be used to select the Yocto distro and skip selection menu.
- ``MACHINE``: Can be used to select the Yocto machine and skip selection menu.
- ``ACCEPT_FSL_EULA``: Can be used to set the Freescale/NXP EULA status and skip
selection menu if applicable).
- ``LICENSE_FLAGS_ACCEPTED``: Can be use to allow e.g. commercial licenses in the resulting images.
- ``UBOOT_ENV_VARS``: Can be use to prepopulate the used u-boot environment (expects a key value list, separated by new lines). To disable pass " "
- ``SCOTTY_FEATURE_LAYERS``: A space separated list of feature layers (alternative to ``scotty setup --features-layers-set`` - CLI flags have always precedence)
Example
^^^^^^^
.. code-block:: bash
$ export ACCEPT_FSL_EULA=1
$ export LICENSE_FLAGS_ACCEPTED=commercial
$ DISTRO=simplecore-distro MACHINE=sm2s-imx8plus scotty setup --features-layers-set distro
$ scotty bitbake core-image-minimal
In addition, it is possible to tweak Scotty behaviour using the environment:
- ``SCOTTY_WHITELIST``: The list of host environment variables exposed in Yocto
environment.
- ``DOCKER_EXTRA_ARGS``: Additional parameters for docker.
Example
.. code-block:: bash
$ export DOCKER_EXTRA_ARGS="-v /home/downloads:/home/scotty/build/downloads -v /home/sstate-cache:/home/scotty/build/sstate-cache"
$ scotty bitbake core-image-minimal
Using Scotty on Windows
-----------------------
Scotty has been tested on Windows with WSL2 and `Docker Desktop`_. It is not
possible to build in a Windows NTFS partition, and you need to build it in an
EXT partition (for example, in your WSL2 home directory).
Building on Windows WSL2+Docker is as fast as building on Linux. However,
Windows can become quite unresponsive:
* The task manager can stop refreshing,
* The terminal may not refresh often,
* Shortcuts like ``Win+R`` and ``Ctrl+Alt+Del`` can trigger several seconds
late.
.. _Docker Desktop: https://docs.docker.com/docker-for-windows/install/
Using Scotty without Docker
---------------------------
If needed, Scotty can be used without docker. In this case, you are in charge of
providing a fully configured environment. This can be done by setting the
``SCOTTY_USE_DOCKER`` environment variable::
$ SCOTTY_USE_DOCKER=0 scotty bitbake core-image-minimal
Scotty Runqemu
==============
``Scotty Runqemu`` is an additional tool allowing to start QEMU to run a QEMU targeted build.
Runqemu Example
---------------
.. code-block:: bash
$ export DISTRO=simplecore-distro MACHINE=qemux86-64
$ scotty setup --features-layers-set examples
$ scotty bitbake simplecore-weston-demo-image
$ scotty-runqemu simplecore-weston-demo-image
scotty-layers.yaml
==================
All the information ``scotty`` uses is defined in ``scotty-layers.yaml`` in the ``manifest`` repository.
This file is a ``yaml`` file containing the following sections
base section
------------
Allowed number of sections in the ``yaml``: 1
This section defines the layers that are **always** used in any setup.
+-------------+--------+----------------------------+--------------------------------------+
| Key | Type | Description | Example |
+=============+========+============================+======================================+
| description | string | Human readable description | description: "My base layers" |
+-------------+--------+----------------------------+--------------------------------------+
| layers | list | Paths to layers to be used | layers: |
| | | | ? "meta-openembedded/meta-oe" |
+-------------+--------+----------------------------+--------------------------------------+
distro_* section
----------------
Allowed number of sections in the ``yaml``: 1..n
These sections define the possible selections for Yocto’s ``DISTRO`` setting.
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| Key | Type | Description | Example |
+===================+========+================================================+=================================================+
| description | string | Human readable description | description: "Super Distro" |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| menu_priority | int | Defines the order in the menu | menu_priority: 100 |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| layers | list | Paths to layers to be used | layers: |
| | | | ? "poky/meta" |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| supported_distros | dict | key/value table of DISTRO name and description | supported_distros: &distros_poky |
| | | | poky: "poky: Yocto Project Reference Distro" |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
machines_* section
------------------
Allowed number of sections in the ``yaml``: 1..n
These sections define the needed layer set for the selected ``MACHINE`` entry.
+-------------------+--------+-----------------------------------------+----------------------------------+
| Key | Type | Description | Example |
+===================+========+=========================================+==================================+
| description | string | Human readable description | description: "My machine layers" |
+-------------------+--------+-----------------------------------------+----------------------------------+
| layers | list | Paths to layers to be used | layers: |
| | | | ? "meta-intel" |
+-------------------+--------+-----------------------------------------+----------------------------------+
| machine_pattern | string | Regular expression matching ``MACHINE`` | machine_pattern: "sm2s-.l" |
+-------------------+--------+-----------------------------------------+----------------------------------+
overlays_* section
------------------
Allowed number of sections in the ``yaml``: 0..n
These sections define the available device tree overlays to be selectable by ``scotty``
+-------------------+--------+-----------------------------------------+----------------------------------+
| Key | Type | Description | Example |
+===================+========+=========================================+==================================+
| description | string | Human readable description | description: "My overlays" |
+-------------------+--------+-----------------------------------------+----------------------------------+
| overlays | dict | key/value table | overlays: |
| | | overlay-filename/description | test.dtb: "my DTB" |
+-------------------+--------+-----------------------------------------+----------------------------------+
| machine_pattern | string | Regular expression matching ``MACHINE`` | machine_pattern: "sm2s-.l" |
+-------------------+--------+-----------------------------------------+----------------------------------+
feature_* section
-----------------
Allowed number of sections in the ``yaml``: 0..n
These sections define the possible selections for additional features.
Features can reference each other by using yaml anchors.
The following example
.. code:: yaml
feature_myfeature:
menu_priority: 200
description: "My feature layer"
layers: &layers_myfeature
<<: [*layers_poky]
? "meta-myfeature"
supported_distros: &distros_myfeature
<<: [*distros_poky]
defines a feature that is called ``My feature layer``. It will add ``meta-myfeature`` to the layer set, and is only applicable
if ``poky`` is selected as a DISTRO. By selecting this feature all layers defined by the ``poky`` DISTRO will be setup as well.
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| Key | Type | Description | Example |
+===================+========+================================================+=================================================+
| description | string | Human readable description | description: "My feature" |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| menu_priority | int | Defines the order in the menu | menu_priority: 100 |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| layers | list | Paths to layers to be used | layers: |
| | | | ? "meta-myfeature" |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| supported_distros | dict | key/value table of DISTRO name and description | supported_distros: &distros_myfeature |
| | | | <<: [\*distros_poky] |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
| add-overlays | object | Additional selectable overlays of the feature | add-overlays: |
| | | see `overlays section` for details | overlays: |
+-------------------+--------+------------------------------------------------+-------------------------------------------------+
scotty-layers.ext.yaml
======================
``scotty`` will additionally scan for ``scotty-layers.ext.yaml`` in the checked out repositories.
These files are written in the same syntax as ``scotty-layers.yaml`` and allow the standard ``scotty-layers.yaml`` to be extended.
The files have to be placed in the root of the bitbake layer.