Magento Functional Testing Framework Changelog ================================================ 2.3.14 ----- ### Enhancements * Maintainability * `command.php` is now configured with an `idleTimeout` of `60` seconds, which will allow tests to continue execution if a CLI command is hanging indefinitely. 2.3.13 ----- ### Enhancements * Traceability * Failed test steps are now marked with a red `x` in the generated Allure report. * A failed `suite` `` now correctly causes subsequent tests to marked as `failed` instead of `skipped`. * Customizability * Added `waitForPwaElementVisible` and `waitForPwaElementNotVisible` actions. * Modularity * Added support for parsing of symlinked modules under `vendor`. ### Fixes * Fixed a PHP Fatal error that occurred if the given `MAGENTO_BASE_URL` responded with anything but a `200`. * Fixed an issue where a test's `` would run twice with Codeception `2.4.x` * Fixed an issue where tests using `extends` would not correctly override parent test steps * Test actions can now send an empty string to parameterized selectors. ### GitHub Issues/Pull requests: * [#297](https://github.com/magento/magento2-functional-testing-framework/pull/297) -- Allow = to be part of the secret value * [#267](https://github.com/magento/magento2-functional-testing-framework/pull/267) -- Add PHPUnit missing in dependencies * [#266](https://github.com/magento/magento2-functional-testing-framework/pull/266) -- General refactor: ext-curl dependency + review of singletones (refactor constructs) * [#264](https://github.com/magento/magento2-functional-testing-framework/pull/264) -- Use custom Backend domain, refactoring to Executors responsible for calling HTTP endpoints * [#258](https://github.com/magento/magento2-functional-testing-framework/pull/258) -- Removed unused variables in FunctionCommentSniff.php * [#256](https://github.com/magento/magento2-functional-testing-framework/pull/256) -- Removed unused variables 2.3.12 ----- ### Enhancements * Fetched latest allure-codeception package 2.3.11 ----- ### Fixes * `mftf run:failed` now correctly regenerates tests that are in suites that were parallelized (`suite` => `suite_0`, `suite_1`) 2.3.10 ----- ### Enhancements * Maintainability * Added new `mftf run:failed` commands, which reruns all failed tests from last run configuration. ### Fixes * Fixed an issue where mftf would fail to parse test materials for extensions installed under `vendor`. * Fixed a Windows compatibility issue around the use of Magento's `ComponentRegistrar` to aggregate paths. * Fixed an issue where an `element` with no `type` would cause PHP warnings during test runs. 2.3.9 ----- ### Fixes * Logic for parallel execution were updated to split default tests and suites from running in one group. 2.3.8 ----- ### Fixes * `ModuleResolver` will now only scan under `MAGENTO_BP/app/code/...` and `MAGENTO_BP/vendor/...` for `/Test/Mftf` directories. * Fixed an issue where `Test.xml` files that did not end with `*Test.xml` would not be scanned for duplicates and other XML validation. 2.3.7 ----- ### Enhancements * Traceability * Test generation errors output xml filename where they were encountered, as well as xml parent nodes where applicable. * Duplicate element detection now outputs parent element where duplicate was found. * Maintainability * Standalone MFTF can now be pointed at a Magento installation folder to generate and execute tests. * See DevDocs for more information. * MFTF now checks for `test` and `actionGroup` elements that have the same `name` in the same file. * Customizability * Updated prefered syntax for `actionGroup` `argument`s that use `xml.data` (old syntax is still supported) * Old: `xml.data` * New: `{{xml.data}}` * Modularity * `ModuleResolver` now utilizes each Magento module's `registration.php` to map MFTF test material directories. ### Fixes * The `waitForPageLoad` action now correctly uses the given `timeout` attribute for all of its checks. * Firefox compatibility issues in javascript error logging were fixed. * Fixed an issue where arguments containing `-` would not properly resolve parameterized selectors. * Fixed an issue where actions using `parameterArray` would not resolve `$persisted.data$` references. * Fixed an issue where composer installations of Magento would fail to parse MFTF materials under a path `vendor/magento/module-/` 2.3.6 ----- ### Enhancements * Maintainability * A `-r` or `--remove` flag has been introduced to `bin/mftf` commands to clear out the contents of the `_generated` folder before generation. This flag has been added to the following commands: * `generate:tests` * `generate:suite` * `run:test` * `run:group` * Customizability * Persisted data handling mechanisms have been reworked. * All persisted data is now referenced with the single `$` syntax (old syntax is still supported): * `$persistedData.field$` * Persisted data resolution now starts in its own scope and broadens if no matching `stepKey` was found in the current scope. * Added support for referencing `suite` persisted data in tests. * Added support for removing data created in between test scopes (`test`, `before/after`, `suite`). * An attribute `skipReadiness` has been added to all test actions, allowing the individual test action to completely bypass the `ReadinessExtension` if it is enabled. ### Fixes * To prevent Allure reporting from collating tests with identical `title`, the `testCaseId` annotation is now automatically prepended to the `title` annotation when tests are generated. * The `magentoCLI` command now correctly removes `index.php` if it is present in the `MAGENTO_BASE_URL`. * Invalid XML errors now indicate which XML file caused the error. * Attempting to `extend` a test that does not exist now skips the generation of the test. * Fixed an issue where a `suite` would generate invalid PHP if the `before` or `after` contained only `createData` actions. * Fixed an issue where a selector inside an `actionGroup` would incorrectly append the `actionGroup`'s `stepKey` to the selector. 2.3.5 ----- ### Fixes * Removed `PageReadinessExtension` from default enabled extensions due to Jenkins instability. 2.3.4 ----- ### Fixes * MagentoWebDriver overrides `parent::_after()` function and remaps to `runAfter()`, necessary to solve compatibility issues in Codeception `2.3.x`. 2.3.3 ----- ### Fixes * Defaults in `etc/config/functional.suite.dist.yml` changed: window-size to `1280x1024`, and removed `--ingonito` flag. 2.3.2 ----- ### Fixes * The `executeJs` `function` no longer escapes persisted variables referenced via `$$persisted.key$$`. * Extending a test no longer fails to generate the parent `test`'s `before`/`after` blocks if the parent was skipped. 2.3.1 ----- ### Enhancements * Maintainability * `mftf build:project` now copies over the `command.php` file into the parent Magento installation, if detected. 2.3.0 ----- ### Enhancements * Traceability * MFTF now outputs generation run-time information, warnings, and errors to an `mftf.log` file. * Overall error messages for various generation errors have been improved. Usage of the `--debug` flag provides file-specific errors for all XML-related errors. * Allure Reports now require a unique `story` and `title` combination, to prevent collisions in Allure Report generation. * The `features` annotation now ignores user input and defaults to the module the test lives under (for clear Allure organization). * The `` annotation has been replaced with a `` annotation, allowing for nested `IssueId` elements. * Tests now require the following annotations: `stories`, `title`, `description`, `severity`. * This will be enforced in a future major release. * Modularity * MFTF has been decoupled from MagentoCE: * MFTF can now generate and run tests by itself via `bin/mftf` commands. * It is now a top level MagentoCE dependency, and no longer relies on supporting files in MagentoCE. * It can be used as an isolated dependency for Magento projects such as extensions. * `generate:tests` now warns the user if any declared `` has an inconsistent `module` (`Backend` vs `Magento_Backend`) * The `--force` flag now completely ignores checking of the Magento Installation, allowing generation of tests without a Magento Instance to be running. * Customizability * Various test materials can now be extended via an `extends="ExistingMaterial"` attribute. This allows for creation of simple copies of any `entity`, `actionGroup`, or `test`, with small modifications. * `test` and `actionGroup` deltas can now be provided in bulk via a `before/after` attribute on the `test` or `actionGroup` element. Deltas provided this way do not need individual `before/after` attributes, and are inserted sequentially. * Secure and sensitive test data can now be stored and used via a new `.credentials` file, with declaration and usage syntax similar to `.env` file references. * A new `` action has been added to allow users to create and use dates according to the given `date` and `format`. * See DevDocs for more information on all above `Customizability` features. * Maintainability * New `bin/mftf` commands have been introduced with parity to existing `robo` commands. * `robo` commands are still supported, but will be deprecated in a future major release. * The `mftf upgrade:tests` command has been introduced, which runs all test upgrade scripts against the provided path. * A new upgrade script was created to replace all test material schema paths to instead use a URN path. * The `mftf generate:urn-catalog` command has been introduced to create a URN catalog in PHPStorm to support the above upgrade. * A warning is now shown on generation if a page's url is referenced without specifying the url (`{{page}}` vs `{{page.url}}`). * An error is now thrown if any test materials contain any overriding element (eg different ``s in a `
` with the same `name`) * This previously would cause the last read element to override the previous, causing a silent but potentially incorrect test addition. * Test distribution algorithm for `--config parallel` has been enhanced to take average step length into account. ### Fixes * `_after` hook of tests now executes if a non test-related failure causes the test to error. * Fixed periods in Allure Report showing up as `•`. * Fixed Windows incompatibility of relative paths in various files. * Suites will no longer generate if they do not contain any tests. * Fixed an issue in generation where users could not use javascript variables in `executeJS` actions. * Fixed an issue in generation where entity replacement in action-groups replaced all entities with the first reference found. * Fixed an issue in generation where `createData` actions inside `actionGroups` could not properly reference the given `createDataKey`. * Fixed an issue where `suites` could not generate if they included an `actionGroup` with two arguments. * Fixed an issue in generation where calling the same entity twice (with different parameters) would replace both calls with the first resolved value. * The `magentoCLI` action now correctly executes the given command if the `MAGENTO_BASE_URL` contains `index.php` after the domain (ex `https://magento.instance/index.php`) * The `stepKey` attribute can no longer be an empty. * Variable substitution has been enabled for `regex` and `command` attributes in test actions. ### GitHub Issues/Pull requests: * [#161](https://github.com/magento/magento2-functional-testing-framework/pull/161) -- MAGETWO-46837: Implementing extension to wait for readiness metrics. * [#72](https://github.com/magento/magento2-functional-testing-framework/issues/72) -- declare(strict_types=1) causes static code check failure (fixed in [#154](https://github.com/magento/magento2-functional-testing-framework/pull/154)) 2.2.0 ----- ### Enhancements * Traceability * Javascript errors are now logged and reported in test output. * Test failures are no longer overwritten by failures in an `` hook. * Tests will no longer execute an `` hook twice if a failure triggered in the `` hook. * Tests marked with `` will now appear in generated Allure reports. * Along with the above, the `robo group` command no longer omits the `skip` group (skipped tests are picked up but not fully executed). * Modularity * MFTF no longer relies on relative pathing to determine its path to tests or Magento (favoring composer information if available). * Tests and test materials are now read in from Magento modules as well as extensions in addition to `dev/tests/acceptance`. * See DevDocs `Getting Started` for details on expected paths and merge order. * Customizability * Creation of Suites is now supported * `` can include tests via `name`, module, or `` tags. * Consolidation of preconditions can be achieved via use of `` tags in a `` * All normal test actions are supported * Data returned from actions is not available for reference in subsequent tests (`createData` or `grab` actions). * `robo generate:tests` generates all suites and tests, and can be given a JSON configuration to generate specific test/suites. * See MFTF Devdocs "Suite" page for more details. * `` may now be called against a `url` instead of a stepKey reference. * `` may now be given an additional `x/y` offset. * `` now returns a variable based on what the executed script returns. * Added `` `type="block"`. * `` elements may now be blank (contain no child sections). * Maintainability * `robo generate:tests --config parallel` now accepts a `--lines` argument, for grouping and sorting based on test length. * `robo generate:tests` now checks for: * Duplicate step keys within an `actionGroup`. * Ambiguous or invalid `stepKey` references (in merge files). * `robo generate:tests` now suppresses warnings by default. The command now accepts a `--verbose` flag to show full output including warnings. ### Fixes * Exception message for the `` action now correctly references the `selector` given. * Usage of multiple parameterized elements in a `selector` now correctly resolves all element references. * Usage of multiple uniqueness references on the same entity now generate correctly. * Persisted entity references are correctly interpolated with `` url of `type="admin"`. * Metadata that contains 2 or more params in its `url` now correctly resolve parameters. * Arguments can now be passed to `x` and `y` attributes in `actionGroup`. * Arguments can now be passed to nested `` action elements. * The `` action can now be used to assert against empty strings. * Empty `` elements within an `` now generate correctly. * Mapping of the `` to the custom command has been fixed. ### GitHub Issues/Pull requests: * [#89](https://github.com/magento/magento2-functional-testing-framework/pull/89) -- Add ability to use array entities as arguments. * [#68](https://github.com/magento/magento2-functional-testing-framework/issues/68) -- Excessive double quotes are being generated in WaitForElementChange method arguments (fixed in [#103](https://github.com/magento/magento2-functional-testing-framework/pull/103)) * [#31](https://github.com/magento/magento2-functional-testing-framework/issues/31) -- Can't run tests without a store having "default" store code (fixed in [#86](https://github.com/magento/magento2-functional-testing-framework/pull/86)) 2.1.2 ----- ### Enhancements * Added support for PHP version 7.2 2.1.1 ----- ### Enhancements * Modularity * MFTF now supports the existence of tests as composer package dependencies or as living alongside core modules. Supported paths: * `magento2ce/vendor/[vendor]/[module]/Test/Acceptance` * `magento2ce/app/code/[vendor]/[module]/Test/Acceptance` * Maintainability * Robo command `generate:tests` now accepts a `--nodes` argument that specifies the number of test manifest files to generate for parallel configuration. ### Fixes * Data returned by `grab` and `createData` actions can now be used in `` test steps by their `stepKey` reference. * Fixed an issue where `` elements inside `` data xml would overwrite one another when merged. * Fixed an issue where `` elements inside `` metadata xml would overwrite one another when merged. * Nested assertion syntax now correctly allows for values passed in to resolve `{{entity.data}}` references. * Test action `` now correctly resolves entity references passed in to `filterSelector` and `optionSelector`. * The robo command `generate:tests --force` no longer requires a `MAGENTO_BASE_URL` to be defined in the `.env` file. * All `feature` and `story` annotations now live under in the method and not class level in output test php. * This is to work around a bug with the Allure-Codeception adapter version `1.2.6`, which was a dependency update in MFTF `2.1.0`. 2.1.0 ----- ### Enhancements * Traceability * Severity in `` tags now properly reflect Magento severity values. * Modularity * Added ability to pass in simple values to actionGroups via addition of `type` attribute in actionGroup `` declaration. * For examples, see devdocs article on actionGroups. * Merging resolution now depends on Magento instance's installed modules. This also means merging order now follows the expected module merging order. * Customizability * Added `` action. This action takes in an array of data and asserts that the array is properly sorted, according to the provided `sortOrder` * Added `` action. This is a variation of `` that is given a `filterSelector`, `optionSelector`, and an `` of options to select. * For a working sample, see `SearchAndMultiselectActionGroup.xml` under `Catalog` in magento2ce. * Test actions that deal with `` now utilize and grab the page's full url, not just the `/path?query#fragment` portion. * All `` actions support a clearer, more readable nested syntax. * Both old and new syntax are supported. See devdocs `Assertions` article for examples. * Added support for overriding a data-entity's `field` values during test runtime, prior to persistence via `` actions. * Added `removeBackend="true"` attribute to ``. Only applicable to `operation` definitions of `type="adminFormKey"`, attribute prevents pre-pending of `MAGENTO_BACKEND_NAME` to the `url` specified. * Specific to use case where `adminFormKey` operations don't follow `MAGENTO_BASE_URL/MAGENTO_BACKEND_NAME/MAGENTO_BACKEND_NAME/API_URL` format. * Readability * Data Entities used in tests (``, ``, `
`, ``, ``) now require alphanumeric naming. * Maintainability * Documentation for all test actions have been added to XML schema. Turning on documentation hinting will display relevant information while writing test XML. * All references to `.env` file contents are now resolved at test runtime, as opposed to generation. ### Fixes * Fixed an issue with using the character `-` in parameterized selector references. * Users should now be able to use any characters except for `'` when providing a `'stringLiteral'` to a parameterized selector/url. * Fixed an issue where entity substitution was not enabled in all `` test actions. ### GitHub Issues/Pull requests: * [#37](https://github.com/magento/magento2-functional-testing-framework/issues/37) -- Unable to make API requests using self signed certificate to HTTPS domain (fixed in [#39](https://github.com/magento/magento2-functional-testing-framework/pull/39)) 2.0.3 ----- ### Enhancements * Readability * Added the ability to refer to `custom_attribute` data in persisted entities via `key` instead of index. * ex. `url_key` in category entity: `$category.custom_attributes[3][value]$` and `$category.custom_attributes[url_key]$` are both valid. * Maintainability * Added check for duplicate `stepKey` attributes at test generation. This check is scoped to `` tags within a single `` tag in a single file. ### Fixes * Fixed inability to use `` with `` in test hooks. * Fixed inability to use `0` as data in an entity. * Fixed an issue where using `` tag of `` would cause test generation failures. * Fixed an issue where the `` action could not be used twice in in a ``. * Fixed an issue where specifying duplicate test actions in test delta's would cause generation errors. * Fixed an issue where test failure screenshots were being taken at the end of the test hook, as opposed to at the point of failure. * Operation `metadata` with an `auth` of type `adminFormKey` will now automatically append specified `MAGENTO_BACKEND_NAME` if necessary. 2.0.2 ----- ### Enhancements * Customizability * Added the `` test action. Action takes the given `command=""` and passes it for execution in Magento Environment. * Note: Installation step to enable above action has been added. See `Step 5` in the MFTF `Getting Started` article. * Maintainability * Tests now run actions declared in `` hook in both successful and failed test runs. ### Fixes * Fixed inability to use `[]` characters within selector/url parameters. * Fixed a bug where the `` action did not return a variable for test use. * Fixed a bug where the `` action could not be used twice in an ``. 2.0.1 ----- ### Fixes * Fixed an issue with `group` annotation. 2.0.0 ----- ### Enhancements * Modularity * Replaced the `` test action with the action group `LoginAsAdmin`. * Added the `.env` file variable `CUSTOM_MODULE_PATHS` which can be used to point to any custom extensions that you may want to write tests against. * Added the `` property to distinguish between admin and storefront. * Added support for `SectionName.elementName` references in any `function` attributes. * Customizability * Changed page objects where `area="admin"` to prepend the `MAGENTO_BACKEND_NAME` value from the `.env` file. * Added support for HTTP requests that do not require authentication. * Readability * Renamed `` XML root nodes to match the content they contain, e.g. `` or ``. * Renamed all instances of the word *Cest* with *Test*. The *Cest* name will no longer be used in the MFTF project. * Maintainability * Removed the `returnVariable` property from any test actions that return values. Instead, the `stepKey` property will be used as the name of the variable and be referenced as before. ### Fixes * Fixed the `unselectOption.parameterArray` property to work as expected. * Fixed a crash if you had a system environment variable set with the same name as any variable in the `.env` file. * Fixed any actions that refer to *CurrentUrl*, such as ``, to now look at the full webdriver address. * Fixed the `` test action to not assume that you always want to dismiss UI notifications. 1.0.0 ------ ### Core features * **Traceability** for clear logging and reporting capabilities * **Modularity** to run tests based on modules/extensions installed * **Customizability** to have an ability to customize existed tests * **Readability** using clear declarative XML test steps * **Maintainability** based on simple test creation and overall structure ### Supported systems #### Operation systems * Windows 10 * macOS Sierra #### Browser * Chrome (Latest) with ChromeDriver Latest ### Known issues * Support for Firefox is currently incomplete. This will be resolved to support Firefox 57 (Quantum) and latest Gecko driver in next minor release. * `MAGENTO_BASE_URL` in _.env_ file must have `/` at the end. Example: http://magento.com/