Traffic Ops

Introduction

Traffic Ops uses a Postgres database to store the configuration information, and the Mojolicious framework to generate the user interface and REST APIs.

Software Requirements

To work on Traffic Ops you need a *nix (MacOS and Linux are most commonly used) environment that has the following installed:

• Carton 1.0.12
  • cpan JSON
  • cpan JSON::PP
• Go 1.8.3
• Perl 5.10.1
• Git
• Postgres 9.6.6
• Goose

Addionally, the installation of the following RPMs (or equivalent) is required:

• All RPMs listed in Traffic Ops - Migrating from 1.x to 2.x

Traffic Ops Project Tree Overview

/opt/traffic_ops/app

• bin/ - Directory for scripts, cronjobs, etc
• conf/
  • /development - Development (local) specific config files.
  • /misc - Miscellaneous config files.
  • /production - Production specific config files.
  • /test - Test (unit test) specific config files.
• db/ - Database related area.
  • /migrations - Database Migration files.
• lib/
  • /API - Mojo Controllers for the /API area of the application.
  • /Common - Common Code between both the API and UI areas.
  • /Extensions
  • Fixtures/ - Test Case fixture data for the ‘to_test’ database. * /Integration - Integration Tests.
  • /MojoPlugins - Mojolicious Plugins for Common Controller Code.
  • Schema/ - Database Schema area. * /Result - DBIx ORM related files.
  • /Test - Common Test.
  • /UI - Mojo Controllers for the Traffic Ops UI itself.
  • Utils/ * /Helper - Common utilities for the Traffic Ops application.
• log/ - Log directory where the development and test files are written by the app.
• public/

• css/ - Stylesheets.
• images/ - Images.
• js/ - Javascripts

• script/ - Mojo Bootstrap scripts.
• t/ - Unit Tests for the UI.

• api/ - Unit Tests for the API.

• t_integration/ - High level tests for Integration level testing.
• templates/ - Mojo Embedded Perl (.ep) files for the UI.

Perl Formatting Conventions

Perl tidy is for use in code formatting. See the following config file for formatting conventions.

edit a file called $HOME/.perltidyrc

-l=156
-et=4
-t
-ci=4
-st
-se
-vt=0
-cti=0
-pt=1
-bt=1
-sbt=1
-bbt=1
-nsfs
-nolq
-otr
-aws
-wls="= + - / * ."
-wrs=\"= + - / * .\"
-wbb="% + - * / x != == >= <= =~ < > | & **= += *= &= <<= &&= -= /= |= + >>= ||= .= %= ^= x="

Database Management

The admin.pl script is for use in managing the Traffic Ops database tables. Below is an example of its usage.

$ db/admin.pl

Usage: db/admin.pl [–env (development|test|production)] [arguments]

Example: db/admin.pl --env=test reset

Purpose: This script is used to manage the database. The environments are defined in the dbconf.yml, as well as the database names.

• To use the admin.pl script, you may need to add traffic_ops/lib and traffic_ops/local/lib/perl5 to your PERL5LIB environment variable.

Arguments	Description
create	Execute db ‘create’ the database for the current environment.
down	Roll back a single migration from the current version.
drop	Execute db ‘drop’ on the database for the current environment.
redo	Roll back the most recently applied migration, then run it again.
reset	Execute db drop, create, load_schema, migrate on the database for the current environment.
seed	Execute SQL from db/seeds.sql for loading static data.
setup	Execute db drop, create, load_schema, migrate, seed on the database for the current environment.
status	Print the status of all migrations.
upgrade	Execute migrate then seed on the database for the current environment.

Installing The Developer Environment

To install the Traffic Ops Developer environment:

1. Clone the traffic_control repository from github.com.
2. Install the local dependencies using Carton (cpanfile).

$ cd traffic_ops/app
$ carton

3. Set up a role (user) in Postgres

   See Postgres instructions on initdb https://wiki.postgresql.org/wiki/First_steps

4. Enter db/admin.pl --env=<enviroment name> setup to set up the traffic_ops database(s).

   • Unit test database: $ db/admin.pl --env=test setup
   • Development database: $ db/admin.pl --env=development setup
   • Integration database: $ db/admin.pl --env=integration setup
   
   

   Running the the admin.pl script in setup mode should look like this:

   master $ db/admin.pl --env=development setup
   Using database.conf: conf/development/database.conf
   Using database.conf: conf/development/database.conf
   Using database.conf: conf/development/database.conf
   Using database.conf: conf/development/database.conf
   Using database.conf: conf/development/database.conf
   Using database.conf: conf/development/database.conf
   Executing 'drop database to_development'
   Executing 'create database to_development'
   Creating database tables...
   Warning: Using a password on the command line interface can be insecure.
   Migrating database...
   goose: migrating db environment 'development', current version: 0, target: 20150210100000
   OK    20141222103718_extension.sql
   OK    20150108100000_add_job_deliveryservice.sql
   OK    20150205100000_cg_location.sql
   OK    20150209100000_cran_to_asn.sql
   OK    20150210100000_ds_keyinfo.sql
   Seeding database...
   Warning: Using a password on the command line interface can be insecure.
   

5. (Optional) To load temporary data into the tables: $ perl bin/db/setup_kabletown.pl

6. Run the postinstall script: traffic_ops/install/bin/postinstall

7. To start Traffic Ops, enter $ bin/start.pl

   The local Traffic Ops instance uses an open source framework called morbo, starting following the start command execution.

   Start up success includes the following:

[2015-02-24 10:44:34,991] [INFO] Listening at "http://*:3000".

Server available at http://127.0.0.1:3000.

8. Using a browser, navigate to the given address: http://127.0.0.1:3000
9. For the initial log in:

• User name: admin
• Password: password

10. Change the log in information.

Test Cases

Use prove to execute test cases. Execute after a carton install:

• To run the Unit Tests: $ local/bin/prove -qrp  t/
• To run the Integration Tests: $ local/bin/prove -qrp t_integration/

The KableTown CDN example

The integration tests will load an example CDN with most of the features of Traffic Control being used. This is mostly for testing purposes, but can also be used as an example of how to configure certain features. To load the KableTown CDN example and access it:

1. Run the integration tests
2. Start morbo against the integration database: export MOJO_MODE=integration; ./bin/start.pl
3. Using a browser, navigate to the given address: http://127.0.0.1:3000
4. For the initial log in:

• User name: admin
• Password: password

Extensions

Traffic Ops Extensions are a way to enhance the basic functionality of Traffic Ops in a custom manner. There are three types of extensions:

1. Check Extensions

These allow you to add custom checks to the “Health->Server Checks” view.

2. Configuration Extensions

These allow you to add custom configuration file generators.

3. Data source Extensions

These allow you to add statistic sources for the graph views and APIs.

Extensions are managed using the $TO_HOME/bin/extensions command line script. For more information see Managing Traffic Ops Extensions.

Check Extensions

In other words, check extensions are scripts that, after registering with Traffic Ops, have a column reserved in the “Health->Server Checks” view and that usually run periodically out of cron.

It is the responsibility of the check extension script to iterate over the servers it wants to check and post the results.

An example script might proceed by logging into the Traffic Ops server using the HTTPS base_url provided on the command line. The script is hardcoded with an auth token that is also provisioned in the Traffic Ops User database. This token allows the script to obtain a cookie used in later communications with the Traffic Ops API. The script then obtains a list of all caches to be polled by accessing Traffic Ops’ /api/1.1/servers.json REST target. This list is walked, running a command to gather the stats from that cache. For some extensions, an HTTP GET request might be made to the ATS astats plugin, while for others the cache might be pinged, or a command run over SSH. The results are then compiled into a numeric or boolean result and the script POSTs tha result back to the Traffic Ops /api/1.1/servercheck/ target.

A check extension can have a column of ‘s and ‘s (CHECK_EXTENSION_BOOL) or a column that shows a number (CHECK_EXTENSION_NUM).A simple example of a check extension of type CHECK_EXTENSION_NUM that will show 99.33 for all servers of type EDGE is shown below:

Script here.

Check Extension scripts are located in the $TO_HOME/bin/checks directory.

Currently, the following Check Extensions are available and installed by default:

Cache Disk Usage Check - CDU

This check shows how much of the available total cache disk is in use. A “warm” cache should show 100.00.

Cache Hit Ratio Check - CHR

The cache hit ratio for the cache in the last 15 minutes (the interval is determined by the cron entry).

DiffServe CodePoint Check - DSCP

Checks if the returning traffic from the cache has the correct DSCP value as assigned in the delivery service. (Some routers will overwrite DSCP)

Maximum Transmission Check - MTU

Checks if the Traffic Ops host (if that is the one running the check) can send and receive 8192 size packets to the ip_address of the server in the server table.

Operational Readiness Check - ORT

See Configuring Traffic Server for more information on the ort script. The ORT column shows how many changes the traffic_ops_ort.pl script would apply if it was run. The number in this column should be 0.

Ping Check - 10G, ILO, 10G6, FQDN

The bin/checks/ToPingCheck.pl is to check basic IP connectivity, and in the default setup it checks IP connectivity to the following:

10G

Is the ip_address (the main IPv4 address) from the server table pingable?

ILO

Is the ilo_ip_address (the lights-out-mangement IPv4 address) from the server table pingable?

10G6

Is the ip6_address (the main IPv6 address) from the server table pingable?

FQDN

Is the Fully Qualified Domain name (the concatenation of host_name and . and domain_name from the server table) pingable?

Traffic Router Check - RTR

Checks the state of each cache as perceived by all Traffic Monitors (via Traffic Router). This extension asks each Traffic Router for the state of the cache. A check failure is indicated if one or more monitors report an error for a cache. A cache is only marked as good if all reports are positive. (This is a pessimistic approach, opposite of how TM marks a cache as up, “the optimistic approach”)

Configuration Extensions

NOTE: Config Extensions are Beta at this time.

Data Source Extensions

Traffic Ops has the ability to load custom code at runtime that allow any CDN user to build custom APIs for any requirement that Traffic Ops does not fulfill. There are two classes of Data Source Extensions, private and public. Private extensions are Traffic Ops extensions that are not publicly available, and should be kept in the /opt/traffic_ops_extensions/private/lib. Public extensions are Traffic Ops extensions that are Open Source in nature and free to enhance or contribute back to the Traffic Ops Open Source project and should be kept in /opt/traffic_ops/app/lib/Extensions.

Extensions at Runtime

The search path for extensions depends on the configuration of the PERL5LIB, which is preconfigured in the Traffic Ops start scripts. The following directory structure is where Traffic Ops will look for Extensions in this order.

Extension Directories

PERL5LIB Example Configuration:

export PERL5LIB=/opt/traffic_ops_extensions/private/lib/Extensions:/opt/traffic_ops/app/lib/Extensions/TrafficStats

Perl Package Naming Convention

To prevent Extension namespace collisions within Traffic Ops all Extensions should follow the package naming convention below:

Extensions::<ExtensionName>

Data Source Extension Perl package name example Extensions::TrafficStats Extensions::YourCustomExtension

TrafficOpsRoutes.pm

Traffic Ops accesses each extension through the addition of a URL route as a custom hook. These routes will be defined in a file called TrafficOpsRoutes.pm that should live in the top directory of your Extension. The routes that are defined should follow the Mojolicious route conventions.

Development Configuration

To incorporate any custom Extensions during development set your PERL5LIB with any number of directories with the understanding that the PERL5LIB search order will come into play, so keep in mind that top-down is how your code will be located. Once Perl locates your custom route or Perl package/class it ‘pins’ on that class or Mojo Route and doesn’t look any further, which allows for the developer to override Traffic Ops functionality.