Skip to main content

Test a module

smf_pipeline has a library for the tests of modules, package:smf_pipeline/testing.dart, with three tools. The contract harness generates every app that matters for a module in memory and checks it against the rules of the module model. ModulePackage checks what the package of a module imports and depends on, and roleClassNameProblems checks that its code takes the names of the classes of roles from the roles. The built-in modules test themselves with the harness and ModulePackage, and the tests of the CLI run roleClassNameProblems over all of them. Use all three for yours. A package that defines a role has a fourth, roleNoteProblems, for the note of the role in the guide for coding agents.

The contract harness​

test/counter_module_test.dart
import 'package:smf_bloc/smf_bloc.dart';
import 'package:smf_counter/smf_counter.dart';
import 'package:smf_flutter_core/smf_flutter_core.dart';
import 'package:smf_gen_l10n/smf_gen_l10n.dart';
import 'package:smf_get_it/smf_get_it.dart';
import 'package:smf_go_router/smf_go_router.dart';
import 'package:smf_pipeline/smf_pipeline.dart';
import 'package:smf_pipeline/testing.dart';
import 'package:smf_riverpod/smf_riverpod.dart';
import 'package:smf_shared_preferences/smf_shared_preferences.dart';
import 'package:test/test.dart';

void main() {
  test('every app of the module follows the rules of its roles', () async {
    final harness = ContractHarness(
      ModuleRegistry(const [
        FlutterCoreModule(),
        GoRouterModule(),
        GetItModule(),
        BlocModule(),
        RiverpodModule(),
        // The localization, which the counter uses for the label of its
        // tab, and the preferences that the localization requires.
        GenL10nModule(),
        SharedPreferencesModule(),
        CounterModule(),
      ]),
    );
    final results = await harness.checkAll();
    for (final result in results) {
      expect(result.errors, isEmpty, reason: '${result.contractCase}');
    }
  });
}

The registry has a module for the localization, which the counter uses for the label of its tab, and one for the preferences, which the localization requires. Without them the harness builds no app of the counter with that role, and a counter that lists the role and forgets to give it the label passes. With them the test reports the mistake: "The module gives the router role the text "label" in its data, but not the localization role, so the app would show it in English in every language."

checkAll() builds a case for every module and every role of the registry. For a module, it builds an app for every combination of a provider of the role of its variants and a provider of each role it requires. It repeats this for each subset of the roles that the module only uses, the largest first, with a provider of each role of the subset in the combination too. A combination that would give the app two providers of a role that takes one is left out, such as one with another router for a module that depends on go_router. When no combination for a subset can be in an app with the module, the harness keeps them all, and their cases report why.

A case is named after its module, the provider of each role that has several in the registry, and the roles of its subset. For the counter above, the cases are counter (bloc) and counter (riverpod), without the localization that it only uses, and counter (bloc) with localization and counter (riverpod) with localization. If the counter also used the analytics role, and the registry had two providers of it, one of its cases would be counter (bloc, firebase_analytics) with analytics.

It also builds a case of the module with each provider in the registry that brings a package which the module adds, itself or in a variant, when the two can be in one app (see the packages of a provider). The case is named <module> with <provider>, such as banner with bloc for a module without variants that adds flutter_bloc. When the provider provides another role than that of the module's variants, the case is <module> (<provider of the variant>) with <provider>, one for each variant that adds the package, or for the first variant when only the module itself adds it. A case whose app another case builds already is left out.

For each case, the harness runs the stages of the pipeline up to rendering, the way a run without a terminal that skips external setup would. It answers the questions of the roles like a user who presses Enter, and renders the app in memory. Then it checks, among other things:

  • the rules of the module's kind, the access rules of sockets and roles, and the roles in when;
  • the module rules of every role, and its structural rules over the parsed Dart files of the app;
  • that every provider generates the symbols its role requires;
  • the tags of the sockets in the templates, and templates that mustache would copy with their braces;
  • that every import finds its file and every imported package is a dependency of the app;
  • that a module imports only the files it may use: its own, those of the modules it depends on directly and those of its roles. The imports of a template count for the module of the template, and an import that the pipeline adds for a fragment counts for the module of the fragment, as the errors say with "in the template of …" and "for a fragment of …";
  • that only what a render hook of a role gives imports the files of the modules that give the role data, such as the screens that a router routes: a fragment of the hook, or a file that the hook generates. That holds for the template of a role and for its providers. The template of a brick gets no such allowance, and neither does a contribution for a fragment of its own, since both are the same whatever the modules give the role;
  • that a module which imports or exports the package of a provider of a role in the app, in a template or for a fragment, adds that package itself, even when it depends on the provider. A provider of a role may import, for a fragment or in a file that its render hook generates, the package that a module which gives the role data adds, as the data may need it;
  • that no file of the app, of a brick or of a render hook, is at a path where code generation or the localizations of Flutter write their files once the app is rendered;
  • that each path that the guide for coding agents of the app names, below a top-level directory of a Flutter project or to a Dart file, is a file or a directory of the app.

The files that a render hook generates are files of the app like those of the bricks, and the harness checks them the same way.

What the registry holds decides what the harness can check, so put these modules into it:

  • FlutterCoreModule, the app entry that every app needs;
  • every module that your module depends on, which the registry requires anyway;
  • every provider of a role your module requires or uses that you want it checked with, since the harness builds an app with each, such as SharedPreferencesModule for a module that keeps a setting, SettingsModule for one with a row on the settings screen and GenL10nModule for one with texts. Such a provider needs what it requires in the registry too: SettingsModule a router, such as GoRouterModule, and GenL10nModule the preferences. Add a setting has such a registry;
  • every provider that your module has a variant for, which the registry requires as well;
  • the provider of every package that your module adds and that a provider of a role brings, such as RiverpodModule for a module that adds flutter_riverpod.

SMF allows the package of a provider only in a variant for that provider or with a dependency on it, and smf create checks this only in apps that have the provider. With the provider in the registry, checkAll() builds such an app of your module, so your tests find the mistake rather than a user of your module.

harness.uncheckedProviders() lists, one line each, the providers that no case checks a module with. Each is a provider of a role that a module of the registry requires or uses, and it can be in an app with the module, but no app of the module's cases has it as the provider of that role. That happens when the apps with the provider do not resolve, such as when no module of the registry provides a role that the provider requires. Expect the list to be empty, as the tests of the CLI do. Each module then stays checked with every provider of its roles, also when a role gets another provider:

expect(await harness.uncheckedProviders(), isEmpty);

The modes of a role​

A role can have a mode option, each value of which makes another app, as --auth-mode of the auth role does. For such a role, checkAll() builds one more case of each provider of the role for each value of the option after the first. It is the first case of the provider, which has every role that the role uses, with the value among the options of the case, and its name ends with --<option>=<value>. With firebase_auth and a router in the registry, these are auth by firebase_auth with router --auth-mode=guest and auth by firebase_auth with router --auth-mode=anonymous. The other cases leave the option out, so their apps have its first value.

So the tests of a provider of such a role render its hook for every value, and a second provider gets these cases without a change of the tests. The harness reads the options from the roles of its registry and names no role itself.

In every app with such a role that it renders, the harness makes the choices twice: in a terminal, where it answers as a user who presses Enter, and without one, as smf create --no-input makes them. It reports a role:

  • whose choice with a value of the option is not that value, or whose choice without the option is not the first value;
  • whose two choices differ;
  • that fails or asks in the run without a terminal.

It also reports a value that a case gives an option which does not take it, since the command line would refuse it.

harness.casesOfModes() has the cases of the apps with every module, those of casesOfAll(), once more for each other combination of the values of the mode options of the roles of each app. SMF's CI generates apps from them; see Contributing. An option that you give every case, with ContractHarness(registry, roleOptions: {...}), has that value in every app and no cases of its own.

Checking one app​

harness.check(case) builds one app and returns its result, whose app holds the rendered files:

final result = await ContractHarness(registry).check(
  const ContractCase('counter', requested: [CounterModule.id, BlocModule.id]),
);
final screen = result.app!.files['lib/features/counter/counter_screen.dart']!;

ContractCase takes the modules to request, the provider to pick for a role with several, and the values of role options in roleOptions, such as {'start': '/counter'} or {'auth-mode': 'guest'}. Check the rendered code as code rather than as text, by parsing it with the analyzer and looking for the declarations and calls you expect.

Your module does not know which module provides a role, and its tests should not either. So they check what the module gives a role through the role, and not in the files that the provider of the role renders: lib/core/router/app_router_factory.dart of the router, lib/core/di/dependencies.dart of the DI container, lib/core/layout/app_shell.dart of the layout, lib/core/l10n/l10n.dart and the files of the texts, such as the ARB files of gen_l10n, the screen of the module that provides the settings screen, and lib/bootstrap.dart, lib/main.dart and lib/app.dart, into which the provider of the app entry renders the start-up and the root of the app. Another provider renders them differently. The result gives two views of the app that do not depend on the provider:

  • result.hook is the RoleHookRequest of the app: the data of the roles, the roles that are present, the context of the harness and the choices of the roles, as the hooks of the roles got them when the harness rendered the app. A role builds the input of its hooks from it with hookInput, so a test reads the data as every provider of the role gets it. diRole.graphOf(diRole.hookInput(result.hook!)) has the registrations of the services, routerRole.startIn(routerRole.hookInput(result.hook!)) the route that the router role chose to start the app on, and layoutRole.destinationsIn(layoutRole.hookInput(result.hook!)) the destinations of the main navigation. In the same way, settingsScreenRole.entriesIn has the entries of the settings screen, in the order the screen shows them, localizationRole.textsIn the texts of the app with their getters, localizationRole.localesIn its languages, and authRole.modeIn the sign-in mode that the auth role chose.
  • result.app!.socketOrders has the contributions of every socket that got any, in the order that the pipeline rendered them into the tags of the socket, including the fragments of the render hooks. What the modules and the templates of the roles put into the platform phase of the start-up is in socketOrders[AppEntryRole.bootstrapPlatform], whichever module renders bootstrap().
final registrations =
    diRole.graphOf(diRole.hookInput(result.hook!)).registrations;
final platform = result.app!.socketOrders[AppEntryRole.bootstrapPlatform]!;
final startUp = [
  for (final collected in platform.contributions)
    (collected.contribution as SocketContribution).fragment!.code,
];

result.app!.entriesOf(socket) gives what a keyed socket got, in the order the socket got it: the contributor, the key and the value of each entry. With it a test reads the notes of the guide for coding agents, each with the heading of its section, without reading AGENTS.md. This finds the note of the preferences role in the app of a result:

final ofRole = [
  for (final (origin, _, note)
      in result.app!.entriesOf(AppEntryRole.agentSections))
    if (origin == const RoleTemplateOrigin(preferencesRole)) note.text,
].single;

The tests of the built-in modules check where the note of their module is: in the section of its role, after the note of the role, or in a section of its own. For the names of code that a note tells agents to use, they check that the note gives each name in inline code and that a file of the app declares or uses it, with DartFileIndexer of package:smf_pipeline/testing.dart. What a sentence of a note says about the code, only a test of that code can pin.

When a test compares the app of your module with the app without it, the files of the providers of its roles differ too, since they render the data of your module. Tell those files apart by their owner rather than by their paths: a rendered file whose owner is a module among result.resolution!.providersOf(role) belongs to a provider of that role.

A module whose tests need a main navigation, but which is not a feature itself, can declare small features and a layout of its own in test/, as smf_firebase_analytics does in test/support/navigation.dart.

The rules of a module package​

test/architecture_test.dart
import 'package:smf_pipeline/testing.dart';
import 'package:test/test.dart';

void main() {
  test('the package follows the rules of a module package', () {
    expect(
      const ModulePackage(
        'smf_counter',
        dependencies: {'mason'},
        testModules: {
          'smf_bloc',
          'smf_flutter_core',
          'smf_gen_l10n',
          'smf_get_it',
          'smf_go_router',
          'smf_riverpod',
          'smf_shared_preferences',
        },
      ).problems(),
      isEmpty,
    );
  });
}

problems() reads the package in the current directory, which is the package's own when dart test runs, and reports every package and import that breaks a rule:

  • The code in lib/ imports only the module model of smf_contracts, its own files, the public libraries of its dependencies, and libraries of Dart that do not reach the machine, which rules out dart:io.
  • Of the module packages, the tests use only the module model, the package itself, its dependencies, smf_pipeline and the testModules.
  • The package depends on smf_contracts and its dependencies and on nothing else. Its dev dependencies among the module packages are smf_pipeline and the testModules.

The module packages are smf_contracts, smf_pipeline and every package that depends on smf_contracts, whatever its name. problems() recognizes them by their pubspecs, which it finds through the package config that dart pub get writes, in the directory of the package or in the root of its workspace. When the package config is missing, or the pubspec of a package that it has to check, the problems ask you to run dart pub get.

pub.dev resolves the dev dependencies of a package when it analyzes it, so the dev dependencies between module packages must not form a cycle. When two modules would need each other in their tests, one of them tests with a small provider of its own in test/.

The names of the classes of roles​

A role can require a class, such as the AppShell that every layout generates, or the Destination that the template of the layout role generates. A module that generates code which creates such a class takes the name from the role, as in '${LayoutRole.appShell.name}(', so that the generated code follows the role. roleClassNameProblems(roles) reads the package in the current directory and the module packages it depends on, and reports each string in their lib/ that spells out the name of a class of those roles followed by (, with its path and line:

test('the code takes the names of the classes of roles from the roles', () {
  expect(roleClassNameProblems(ModuleRegistry(modules).roles), isEmpty);
});

Here modules are the modules of the registry of your tests. Comments, code outside strings, such as the Destination(...) of a route, and the bundles in lib/bundles/, which the bricks generate, do not count. roleClassNameProblems finds the module packages as ModulePackage does, and asks you to run dart pub get when the package config is missing.

The notes of roles​

The template of a role adds the note of the role to the guide for coding agents, and the note has to hold whichever modules provide the roles. A package that defines such a role checks it with roleNoteProblems, in the apps of the results of the harness:

test('the notes of the roles name nothing of their providers', () async {
  final results = await ContractHarness(ModuleRegistry(modules)).checkAll();
  expect(
    roleNoteProblems(
      results,
      AppEntryRole.agentSections,
      textOf: (note) => note.text,
    ),
    isEmpty,
  );
});

The check holds each note of a role against the providers of every role in those apps, and against the modules that they depend on. The note may have none of their ids or packages. In its code, it may name nothing that only their files declare, and no file that only they generate. What the roles guarantee is free, such as the symbols of their interfaces.

The check knows a module only from the generated files, so it takes a name of Flutter that only files with one package have for a name of that package. Pass such a name in allowed with the reason, as in allowed: {'NavigatorObserver': 'A class of Flutter.'}. Do the same when a line is about a note of smf_contracts, which you cannot change. The check also reports an entry of allowed that excuses nothing.

Beyond the harness​

The harness renders apps but does not build them. To see your module in a real app, generate apps with your command, as Extending SMF shows, with each state manager and each provider your module works with, and run flutter analyze on them. SMF's CI does this for the built-in modules.

A module with a guard keeps the user from every other screen until its guard allows. So a test that starts an app with the module sees only the flow of the guard, unless it first puts what the guard reads into the state that allows, as a test in an app with onboarding does.

A module whose guard asks for an account opens it for such a test by signing an account up before the app starts, as the tests of the apps with sign_in do. A call of appSession made before the start-up waits for it, and the start-up completes after such calls. So the test calls appSession.signUp() before bootstrap(), without awaiting it, and the app starts with that account. It makes the call outside the body of a test, as in a setUpAll: made in the body of a widget test, the call waits in the fake time of that body, and a start-up that runs in real time never completes. Sign-in has more on tests of the session.

A module that remembers a setting has more to show in a running app: that a choice is saved, and that the next launch reads it. Preferences and settings describes both checks, which hold whichever module provides the preferences.

A module whose generated code is plain Dart can go further in its tests. It can write the rendered files to a temporary directory with the packages of the tests, analyze them and run them in the Dart VM. The tests of smf_get_it and smf_event_bus run the generated services with the real get_it and event_bus this way.

The state of a variant​

The state of a screen is close to plain Dart even when the screen is not: a cubit or a notifier uses little of Flutter. So the tests of a feature with variants can run the state of each variant with the Dart SDK alone, as the tests of smf_sign_in do:

  • They write the Dart files of a rendered app to a temporary directory, with stand-ins for what the state uses of Flutter, such as ChangeNotifier and ValueListenable of the foundation library.
  • The package of each state manager is a stand-in too, which exports the package it builds on: flutter_bloc exports package:bloc/bloc.dart, and flutter_riverpod exports package:riverpod/riverpod.dart. Both are plain Dart, and the package of the module has them as dev dependencies.
  • A script of the test imports the state of a screen and no widget, gives the session a service that it scripts, and runs in the Dart VM. The same scenarios run with each variant: a call that is on its way, a failure, a second call, a user who leaves the screen.

So the tests of the package need no Flutter SDK. The widgets need more of Flutter than a stand-in gives, so the analyzer checks them, and they run in the tests of generated apps.