Metadata-Version: 1.0
Name: cl-parsers
Version: 1.0
Summary: Parsers for Cumulus Linux commands
Home-page: UNKNOWN
Author: Cumulus Networks
Author-email: dev-support@cumulusnetworks.com
License: UNKNOWN
Description: # Cumulus Linux Parsers
        
        ## Installation on Cumulus Linux for execution only
        Use dpkg or apt-get to install the debian package.  If this module is not 
        already installed and dpkg is not acceptable, you can
        copy the cl_parsers subdirectory directly to /usr/lib/python2.7/dist-packages/
        (not recommended)
        
        ## Installation for parser development and testing
        ### QA infrastructure for remote access
        Normally it is desired to develop in an environment with remote access
        to multiple switches, which requires an ssh library.
        
        Brief instructions are provided below to use the Cumulus QA infrastructure
        for that ssh access.
        
        If developing on a local switch, it is unnecessary to install the QA
        infra
        
        #### If on a Hydra or server without the QA infra already installed:
        Follow the instructions here if you haven't setup the SSIM venv via bootstrap.py:
        https://wiki.cumulusnetworks.com/pages/viewpage.action?pageId=27100295
        
        #### If on a Hydra or server with the QA infra already installed:
        Source the QA venv:
        
          ```
          . $TESTINFRA/venv/bin/activate
          ```
        
        
        ### Install Parser Tests:  cl_parsers_tests unit test package
        There are a set of unit tests to verify
        that any updated parsers do not break the existing use cases.  These
        are installed via a separate python package cl_parsers_tests
        
        When editing or developing a parser you will need to git clone that
        project and setup a virtualenv that includes this project:
        
        ```
          cd <git_project_path>
          git clone ssh://git@stash.cumulusnetworks.com:7999/~mbrown/cl_parsers_tests.git
        ```
        
        ### Install cl_parsers (this repo)
        
        ```
          cd <git_project_path>
          git clone ssh://git@stash.cumulusnetworks.com:7999/crmc/cl-parsers.git
        ```
        
        ### Create an venv for your development (if not using the QA venv):
        
        ```
          cd <env_dir>
          virtualenv cl_parsers_test
        ```
        
        ### Source your venv (if not using the QA venv):
        
        ```
          cd <env_dir>
          . cl_parsers_test/bin/activate
        ```
        
        ### Add cl_parsers lib to the virtualenv:
        
        ```
          cd <git_project_path>
          pip install cl_parsers
        ```
        
        Note is not recommended to install the package in '-e, --editable' developer mode
        since that method won't create the parsers tree explictly and you cannot
        add additional parser packages like cl-parsers-bcm to the tree
        
        ### Install additional parsers such as cl-parsers-bcm:
        
        ```
          pip install <git_or_local_path_to_cl-parsers-bcm>
        ```
        
        ## Updating the installation on a laptop or server in production (not development)
        
        From a local repo
        
        ```
          pip install -U path/to/cl-parsers
          pip install -U path/to/cl-parsers-bcm
        ```
        
        Directly from Git
        
        ```
          pip install -U git+ssh://git@stash.cumulusnetworks.com:7999/crmc/cl-parsers.git
          pip install -U git+ssh://git@stash.cumulusnetworks.com:7999/crmc/cl-parsers-bcm.git
        ```
        
        ## Using a Parser
        
        In your script the following is required to use cl_parsers on a command:
        
        ```
        from cl_parsers.parse import parse
        
        output = dut.sudo(command)  # or dut.run, or however you collect a cmd output from a device
        parsed_dct = parse(command, output)
        ```
        
        The parser system should be able to find the output via longest match on the
         command string.  e.g.  ```/usr/lib/cumulus/bcmcmd portstat xe20,xe21``` will match parser ```bcmcmd portstat```
        
        
        ## Fixing or Editing a Parser
        Let's say there is a CLI output change and your script is now failing due to a failed parser.
        
        ### Example:
        
        ethtool output changed in CL3.2 to include two new fields:
        
          ```
          Supported FEC modes: Not reported
          ```
        
          ```
          Advertised FEC modes: Not reported
          ```
        
        Script is failing with a parser error on this command:  dell-s6000-22: 'ethtool swp8'
        
        ### Fixing and testing the parser:
        
        
        1. Test and verify that the parse is truly failing (not some other infrastructure problem):
        
            ```
            cd <git_project_path>/cl_parsers_tests
            ./cl_parsers_tests/utilities/quick_parser_test.py dell-s6000-22 'ethtool swp8'
            ```
        
        1. Edit the parser and insert the missing lines.  In this case make sure the regex lines are wrapped in a (..)? since these
            lines will not appear in older code versions.
        
            Parser to edit:
        
            ```cl_parsers/parsers/layer1/ethtool.py```
        
            Add the two new lines to add to the proper location in the regex:
        
            ```
            (\s+Supported \sFEC\smodes:\s+(?P<supp_fec_modes>[0-9a-zA-Z /]+))?
            ```
        
            ```
            (\s+Advertised\sFEC\smodes:\s+(?P<adv_fec_modes>[0-9a-zA-Z /]+))?
            ```
        
            Also add the two new items to the ``` named_groups``` section if one exists.
        
            ```
                             'supp_fec_modes', 'adv_fec_modes',
            ```
        
        1. Verify that the parser is working correctly on the new output after the changes.  Iterate until correct:
        
            ``` cl_parsers_tests$ ./utilities/quick_parser_test.py dell-s6000-22 'ethtool swp8' ```
        
        1. Verify that the new parser changes still work on all the old tests:
        
             ``` cl_parsers_tests$ ./setup.py test ```
        
             If any failures, fix them and repeat the last 2 steps
        
        1. Collect/create a new test for this new output:
        
             ``` cl_parsers_tests$ ./utilities/collect_parser_test_files.py dell-s6000-22 'ethtool swp8' ```
        
        1. Retest to verify all is clean:
        
             ``` cl_parsers_tests$ ./setup.py test ```
        
        1. Commit the changes after everything passes.
        
        ```
            cd <git_project_path>/cl_parsers
            git add cl_parsers/parsers/layer1/ethtool.py
            git commit -m 'Adding FEC lines for new ethtool output'
            git push
        	
            cd <git_project_path>/cl_parsers_tests
            git add tests/parsers/layer1/ethtool/
            git commit -m 'Adding tests for new ethtool output'
            git push
        ```
        
        
        ## Creating New Parsers
        
        ### The faster way:
        1. Start with one of the two parser templates: ```cl_parsers/parsers/template_minimal.py``` (just the parser)
             or ```cl_parsers/parsers/template.py``` (with optional TestParseParticulars test)
        1. Fill in your code to parse the command output to a dictionary.
        1. Make sure that if the parser does not parse correctly, that it returns 'None' instead of a dictionary.
        
        ### The manual way:
        The following are the requirements of a parser module:
        
        1. Stored in a technology grouping master level directory:
          ```cl_parser/parsers/<technology>/```  e.g. ```technology = layer1``` or ```technology = bgp```
        
        1. A file (module) with the parser for one unique command in it:
          ```cl_parser/parsers/<technology>/<parserName>.py```
        
        1. a string definition giving the longest common match of the direct command name of the parser
             For instance:
             ```name = "ethtool -m"```  makes it different from ```ethtool``` and ```ethtool -S``` parsers
              or:  ```name = "bcmcmd portstat"```
        1. a function in that file named 'parse' to parse an output string from the command into a dictionary
              e.g.  ```def parse(command, output):```
        
        1. If an input text does not parse, the function should return a 'None' (null output)
        
        1. Optional:  A 'Particulars' tests defined in the parser if needed to check specific corner
             cases or for proper/matching output from the parser.  This will be a unittest.TestCase class
             definition named TestParseParticulars. TODO: put this in the tests package dir for the 
             module instead
        
        1. In the case of needing to parse a text file, the 'cat' parser will redirect the file to a parser
            named with the name of the text file without the path.
              e.g.  ```/bin/cat  /var/lib/cumulus/porttab``` only needs a parser named ```porttab``` that parses
              the porttab file
        
        
        
        ## Creating Test Files for each parser
        
        ### After creating a new parser, you must create a set of tests for it:
        
        
        There is a generic test in higher module (test_cl_parsers.py) that verifies
        that the parsed dict is created: i.e.
        ```assert parserName(cmd,inputFileText) is not None```
        To use this test, you just have to create a set of dut command output files
        
        #### The fast way:
        
        Use the script ```cl_parsers_tests/utilities/collect_parser_test_files.py <dut> <cmd>``` to collect
        representative dut output data to verify parsers against.  use ```--help``` to see the usage info for that script.
        
        #### The manual way:
        The tests for each parser should look like this:
        
        1. To run the generic parser test, create a directory with multiple files of CL output text to be parsed,
           each with a unique numerical extension.  The path should mirror the parser path, where the last dir is
           the name of the parser file:
           ``` cl_parsers_tests/tests/parsers/<technology>/<parserName>/<dut>_<cmd>_testinput.1, .2 .3, etc... ```
        
        1. Optional:  To run a specific test, like verifying the parsed dict output never changes,
            create a matching output file in json format:
            ``` cl_parsers_tests/tests/parsers/<technology>/<parserName>/<dut>_<cmd>_testoutput.1, .2 .3, etc... ```
        
        
        ## Contributing your New Parser to the testing community
        
        ### Test your Tests:
        After creating a test, make sure all of them pass via: 
        
        ``` 
        (venv):cl_parsers_tests% python setup.py test 
        ```
        
        ### Commit your tests to the repo after all tests are working:
        Send a pull request or request a review and submit your parser and tests:
        
        ```
            cd <git_project_path>/cl_parsers
            git add cl_parsers/parsers/<technology>/<parsername>.py
            git commit -m 'Adding new parser <parsername>'
            git push
            
            cd <git_project_path>/cl_parsers_tests
            git add tests/parsers/<technoloy>/<parsername>/
            git commit -m 'Adding new parser tests for <parsername>'
            git push
        ```
        
        
        
        TODO: Figure out easiest way to automatically get the command string into the parser while not
         depending on a particular 'sudo' ssh handler implementation
        
        
        
        
        
        
Keywords: cl_parsers
Platform: UNKNOWN
