Testing
bulletinbored ships with a zero-dependency test suite — no PHPUnit, no Composer, no Docker. Just plain PHP you run from the CLI.
Quick Start
# Run all tests
php tests/run.php
# Run a specific test file
php tests/run.php DbQuery
php tests/run.php E2eFlow
php tests/run.php PluginManager
php tests/run.php Auth
php tests/run.php Migrator
php tests/run.php Security
php tests/run.php Response
php tests/run.php Markdown
php tests/run.php PluginRouter
php tests/run.php Registration
php tests/run.php Installer
php tests/run.php SecurityHardening
php tests/run.php ContentCrud
php tests/run.php E2eIntegration
php tests/run.php DatabaseIntegrity
php tests/run.php Suggested
php tests/run.php UpdateManager
php tests/run.php PluginTheme
php tests/run.php Upgrade
php tests/run.php UpdateFailureMode
php tests/run.php EmailSecurity
php tests/run.php EndpointAuthorization
php tests/run.php SessionSecurity
php tests/run.php UploadSecurity
# Verbose output
php tests/run.php --verbose
# List all registered tests
php tests/run.php --list
# Source coverage report (line coverage with Xdebug/PCOV, otherwise loaded files)
php tests/run.php --coverage
# Fail the run when overall line coverage is below 60%
php tests/run.php --coverage --min-coverage=60
# Fail the run when a critical area is below 70%
php tests/run.php --coverage --min-area-coverage=70
# Enable the HTTP integration tests explicitly (auto-enabled on non-Windows)
BB_HTTP_TESTS=1 php tests/run.php BootstrapIntegration
--min-coverage=N and --min-area-coverage=N also enable coverage. They use Xdebug or PCOV when available; without a coverage driver they print a warning instead of failing.
Test Structure
tests/
├── harness.php # Test + TestSuite classes (the engine)
├── bootstrap.php # Test environment bootstrap (temp dirs, file-backed SQLite database, test mode)
├── run.php # CLI runner (--verbose, --list, --coverage, --min-coverage, --min-area-coverage)
├── ApiInstallAuthorizationTest.php # api/install.php authorization
├── AtomicFileTest.php # Atomic metadata writes
├── AuthHardeningTest.php # Auth hardening tests
├── AuthTest.php # Auth, permissions, CSRF, AuthZ tests
├── BootstrapIntegrationTest.php # Real-bootstrap smoke test + optional HTTP header/route checks
├── ContentConcurrencyTest.php # Concurrent content writes
├── ContentCrudTest.php # Thread creation, replies, viewing, pagination
├── ContentHardeningTest.php # Content hardening tests
├── CoreZipUpdateTest.php # Core ZIP update pipeline
├── DatabaseIntegrityTest.php # Foreign keys, constraints, counters, soft-delete
├── DatabaseMatrixTest.php # Cross-database compatibility tests
├── DbQuerySecurityTest.php # Query builder security
├── DbQueryTest.php # Query builder tests
├── E2eFlowTest.php # End-to-end flow tests
├── E2eIntegrationTest.php # End-to-end flow tests (user journeys)
├── EmailSecurityTest.php # SMTP injection and email validation
├── EndpointAuthorizationTest.php # HTTP endpoint authorization matrix
├── HelpersTest.php # Helper module tests
├── InstallerTest.php # Installation process tests
├── MarkdownTest.php # Markdown security tests
├── MigrationRecoveryTest.php # Interrupted migration recovery and retry policy
├── MigratorTest.php # Migration engine tests
├── ModerationHandlerTest.php # Moderation handler integration tests
├── ModerationTest.php # Moderation actions tests
├── MySqlDialectTest.php # MySQL dialect behavior
├── PackageInstallerAdversarialTest.php # Adversarial package archives
├── PluginManagerTest.php # Hook system tests
├── PluginRouterTest.php # Plugin route/middleware registration
├── PluginThemeTest.php # Plugin and theme tests
├── PrereleasePolicyTest.php # Prerelease update policy
├── RateLimitTest.php # Rate limiting
├── RawSqlPolicyTest.php # Raw SQL policy
├── RegistrationTest.php # Registration & login tests
├── ReleasePackageTest.php # Release package contents
├── RendererTest.php # Template engine tests
├── ResponseTest.php # Response object + typed Request tests
├── RouteAuthorizationTest.php # Route authorization
├── SecurityFixesTest.php # Security fixes tests
├── SecurityHardeningTest.php # SQL injection, upload security, security headers
├── SecurityTest.php # CSRF rotation, Request, audit log, trusted proxies
├── SessionFreshnessTest.php # Session freshness
├── SessionSecurityTest.php # Session invalidation tests
├── SessionVersionPolicyTest.php # Session version policy
├── SmtpProtocolTest.php # SMTP protocol
├── SuggestedTest.php # Edge case tests
├── TelemetryTest.php # Anonymous install heartbeat (guards, payload, opt-out)
├── UpdateBackupTest.php # Backup integrity and restore
├── UpdateFailureModeTest.php # Update failure modes
├── UpdateManagerTest.php # Update system, ZIP validation, rollback
├── UpgradeCompatibilityTest.php # Upgrade path compatibility
├── UpgradeTest.php # Upgrade pipeline tests
├── UploadSecurityTest.php # Upload security tests
└── VersionConsistencyTest.php # VERSION / config-sample / release-notes consistency
How Tests Are Registered
Most test files define functions named test_*() that return a Test object, then register them at the bottom:
register_tests('test_foo', 'test_bar', 'test_baz');
The runner (tests/run.php) loads all *Test.php files, collects the registered tests, and executes them. Test files should not call run() or exit() — the runner handles that.
Special case: DatabaseMatrixTest.php
DatabaseMatrixTest.php does not use register_tests(). Instead it calls register_database_matrix_tests() at the bottom, which registers each test through $suite->addTestFactory(), not $suite->addTest(). This is because the same test functions are executed multiple times, once per database driver (SQLite, optionally MySQL/MariaDB).
The file also refuses direct execution: running php tests/DatabaseMatrixTest.php exits with an error. It only registers tests, so it must be run through php tests/run.php (optionally with DB_DRIVER=mysql).
Helper functions
Some test files define helper functions that are not registered as tests. These are named without the test_ prefix to avoid confusion:
setup_schema_endpoint(),create_user_endpoint(),create_post_endpoint()inEndpointAuthorizationTest.phpsetup_schema_upload(),create_user_upload(),create_category_upload(),create_thread_upload(),create_upload_sec(),can_access_upload()inUploadSecurityTest.phpsetup_schema_session(),create_user_session()inSessionSecurityTest.php
Harness API (tests/harness.php)
Class Test
Represents a group of assertions. All methods record pass/fail and print results on run().
$t = new Test('My Feature');
// Generic assertion
$t->assert('description', $condition === true);
// Typed assertions
$t->assertEquals('description', $expected, $actual); // === comparison
$t->assertNotEquals('description', $unexpected, $actual);
$t->assertTrue('description', $value);
$t->assertFalse('description', $value);
$t->assertNull('description', $value);
$t->assertNotNull('description', $value);
$t->assertCount('description', $expectedCount, $array);
$t->assertContains('description', $needle, $array);
$t->assertInstanceOf('description', 'ClassName', $object);
// Exception assertions
$t->assertThrows('description', function() { throw new Exception('test'); });
$t->assertNotThrows('description', function() { /* no exception */ });
// Custom predicate: assertThat(string $desc, callable $callable, callable $predicate)
$t->assertThat('value is positive', fn() => $value, fn($v) => $v > 0);
// Run and print results
$t->run();
Class TestSuite
Aggregates multiple Test instances and reports totals.
$suite = new TestSuite();
$suite->addTest($test1);
$suite->addTest($test2);
$suite->run(); // exits with code 1 if any test failed
Writing Tests
Pattern
Each test file defines functions that return a Test object and registers them with register_tests():
<?php
/**
* Feature tests — tests for a specific component.
*/
require_once __DIR__ . '/harness.php';
require_once __DIR__ . '/../lib/YourClass.php';
function test_feature_behavior(): Test
{
$t = new Test('Feature - Behavior');
// Setup
$instance = new YourClass();
// Execute
$result = $instance->doSomething();
// Assert
$t->assert('Method returns expected result', $result === 'expected');
return $t;
}
function test_feature_edge_case(): Test
{
$t = new Test('Feature - Edge Case');
// Test empty input
$result = (new YourClass())->doSomething('');
$t->assert('Handles empty input gracefully', $result === null);
return $t;
}
// Register tests for the runner
register_tests('test_feature_behavior', 'test_feature_edge_case');
The runner (tests/run.php) loads all test files and executes registered tests together. No exit() call needed — the runner handles exit codes.
Database Tests
Use in-memory SQLite for isolated, fast tests:
function test_dbquery_insert(): Test
{
$t = new Test('DbQuery - Insert');
$pdo = new PDO('sqlite::memory:');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$db = new DbQuery($pdo);
// Create test table
$pdo->exec("CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT NOT NULL,
email TEXT
)");
// Test insert
$id = $db->table('users')->insert([
'username' => 'alice',
'email' => 'alice@test.com'
]);
$t->assert('Insert returns positive ID', $id > 0);
// Verify data
$user = $db->table('users')->where('id', $id)->first();
$t->assertEquals('Username matches', 'alice', $user['username'] ?? '');
return $t;
}
Router / E2E Flow Tests
Test complete workflows by setting $_SERVER state and dispatching through the Bulletin\Router:
function test_e2e_thread_lifecycle(): Test
{
$t = new Test('E2E: Thread lifecycle');
// Simulate request
$_SERVER = [
'REQUEST_URI' => '/thread/123',
'REQUEST_METHOD' => 'GET',
'HTTP_HOST' => 'localhost',
'SCRIPT_NAME' => '/index.php',
];
$_GET = [];
$router = new Bulletin\Router();
$router->get('/thread/{id:\d+}', function($params) {
return ['status' => 200, 'body' => 'thread:' . $params['id']];
});
ob_start();
$router->dispatch();
$output = ob_get_clean();
$t->assertEquals('Route matches thread pattern', 'thread:123', $output);
return $t;
}
JSON API Tests
Test automatic JSON encoding for API routes:
function test_json_api_flow(): Test
{
$t = new Test('E2E: JSON API');
$_SERVER = [
'REQUEST_URI' => '/api/threads',
'REQUEST_METHOD' => 'GET',
'HTTP_ACCEPT' => 'application/json',
'HTTP_HOST' => 'localhost',
'SCRIPT_NAME' => '/index.php',
];
$router = new Bulletin\Router();
$router->api()->get('/api/threads', function($params) {
return ['threads' => [], 'total' => 0];
});
ob_start();
$router->dispatch();
$output = ob_get_clean();
$decoded = json_decode($output, true);
$t->assert('Returns valid JSON', $decoded !== null);
return $t;
}
Plugin Manager Tests
Test hook system behavior:
function test_hook_filter(): Test
{
$t = new Test('PluginManager - Filter');
$pm = new PluginManager('/tmp', '/tmp/manifest.json');
// Register filter chain
$pm->addHook('content_filter', function($value) {
return $value . 'A';
});
$pm->addHook('content_filter', function($value) {
return $value . 'B';
});
$result = $pm->filter('content_filter', 'start-');
$t->assertEquals('Filter chains callbacks', 'start-AB', $result);
return $t;
}
Auth Tests
Test password hashing, CSRF, permissions:
function test_password_hashing(): Test
{
$t = new Test('Auth - Password Hashing');
$hash = password_hash('secret', PASSWORD_DEFAULT);
$t->assertTrue('Verify correct password', password_verify('secret', $hash));
$t->assertFalse('Reject wrong password', password_verify('wrong', $hash));
return $t;
}
function test_permissions(): Test
{
$t = new Test('Auth - Permissions');
// Setup in-memory DB with roles (resource.action notation)
$pdo = new PDO('sqlite::memory:');
$pdo->exec("CREATE TABLE roles (id INTEGER PRIMARY KEY, name TEXT, permissions TEXT)");
$pdo->exec("INSERT INTO roles VALUES (1, 'admin', '[\"admin.access\",\"users.ban\"]')");
$pdo->exec("CREATE TABLE users (id INTEGER PRIMARY KEY, username TEXT, role TEXT)");
$pdo->exec("INSERT INTO users (id, username, role) VALUES (1, 'admin', 'admin')");
// AuthZ is the single source of truth (no globals, no user_has_permission())
$authz = new AuthZ($pdo);
$t->assertTrue('Admin has permission', $authz->can(1, 'users.ban'));
return $t;
}
Functional Handler Tests
Test handlers directly by setting up the database state and calling the handler function. This tests the actual authorization logic, not just static code analysis:
function test_download_hidden_thread_guest_forbidden(): Test
{
$t = new Test('Download: guest cannot download from hidden thread');
$pdo = setupDB(); // Creates in-memory SQLite with schema
$authz = new AuthZ($pdo);
App::reset();
App::getInstance()->authz = $authz;
App::getInstance()->pdo = $pdo;
$_SESSION = []; // Guest user
// Create hidden thread and associated upload
$stmt = $pdo->prepare("INSERT INTO threads (id, category_id, user_id, title, status) VALUES (1, 1, 1, 'Hidden', 'hidden')");
$stmt->execute();
$stmt = $pdo->prepare("INSERT INTO uploads (id, thread_id, user_id, filename, original_name, size, mime_type) VALUES (100, 1, 1, 'file.txt', 'file.txt', 100, 'text/plain')");
$stmt->execute();
$threw = false;
try {
handle_download(['id' => 100]);
} catch (\Bulletin\ForbiddenException $e) {
$threw = true;
}
$t->assertTrue('Guest blocked from hidden thread download', $threw);
App::reset();
return $t;
}
Note: For “allowed” cases, the handler calls exit after serving the file. To avoid this, either:
- Don’t create the physical file (handler throws
NotFoundExceptionafter passing access control — this verifies the access check passed) - Clean up any residual files with
@unlink()before the test
Test Count
php tests/run.php --list currently reports 517 tests over 54 files. Of those, 470 test functions are core (tests/*Test.php, 53 files) and plugins/editbored contributes 47 (plugins/editbored/tests/EditboredPluginTest.php).
| File | Registered Tests |
|---|---|
ApiInstallAuthorizationTest.php |
3 |
AtomicFileTest.php |
4 |
AuthHardeningTest.php |
8 |
AuthTest.php |
6 |
BootstrapIntegrationTest.php |
3 |
ContentConcurrencyTest.php |
2 |
ContentCrudTest.php |
14 |
ContentHardeningTest.php |
9 |
CoreZipUpdateTest.php |
6 |
DatabaseIntegrityTest.php |
11 |
DatabaseMatrixTest.php |
9 |
DbQuerySecurityTest.php |
5 |
DbQueryTest.php |
11 |
E2eFlowTest.php |
5 |
E2eIntegrationTest.php |
7 |
EmailSecurityTest.php |
14 |
EndpointAuthorizationTest.php |
14 |
HelpersTest.php |
15 |
InstallerTest.php |
11 |
MarkdownTest.php |
6 |
MigrationRecoveryTest.php |
7 |
MigratorTest.php |
7 |
ModerationHandlerTest.php |
6 |
ModerationTest.php |
11 |
MySqlDialectTest.php |
7 |
PackageInstallerAdversarialTest.php |
5 |
PluginManagerTest.php |
14 |
PluginRouterTest.php |
4 |
PluginThemeTest.php |
22 |
PrereleasePolicyTest.php |
6 |
RateLimitTest.php |
5 |
RawSqlPolicyTest.php |
1 |
RegistrationTest.php |
16 |
ReleasePackageTest.php |
3 |
RendererTest.php |
4 |
ResponseTest.php |
9 |
RouteAuthorizationTest.php |
7 |
SecurityFixesTest.php |
26 |
SecurityHardeningTest.php |
16 |
SecurityTest.php |
7 |
SessionFreshnessTest.php |
7 |
SessionSecurityTest.php |
12 |
SessionVersionPolicyTest.php |
2 |
SmtpProtocolTest.php |
10 |
SuggestedTest.php |
10 |
TelemetryTest.php |
10 |
UpdateBackupTest.php |
14 |
UpdateFailureModeTest.php |
15 |
UpdateManagerTest.php |
22 |
UpgradeCompatibilityTest.php |
3 |
UpgradeTest.php |
4 |
UploadSecurityTest.php |
14 |
VersionConsistencyTest.php |
1 |
| Core total | 470 |
plugins/editbored/tests/EditboredPluginTest.php |
47 |
| Total | 517 |
Each test runs multiple assertions, so the assertion count is larger and depends on the environment. The exact number of registered tests also depends on whether DatabaseMatrixTest.php runs once (SQLite only) or again per service database. Run php tests/run.php --list to enumerate the registered tests for your checkout.
tests/bootstrap.php points db_path at a temporary file (<tempDir>/data/db.sqlite) and uses a file-backed SQLite database, not sqlite::memory:. Individual tests may still create their own in-memory database for isolation.
Integration tests
BootstrapIntegrationTest.php covers the parts of src/bootstrap.php that the
CLI suite skips because of the BULLETIN_TEST_MODE early return:
- a subprocess smoke test boots the real bootstrap (session, UTC timezone,
autoloader, i18n) — always runs; - optional HTTP tests start the PHP built-in server with
router.phpand assert
the real security headers and that sensitive files return403. They run
automatically on non-Windows platforms or whenBB_HTTP_TESTS=1is set (CI
sets it explicitly).
Coverage
php tests/run.php --coverage prints line coverage using Xdebug (executable
lines) or PCOV (executed lines) when available, and otherwise a
dependency-free report of which src/ and lib/ files the suite loaded.
Coverage spans both trees. With a driver, --min-coverage=N fails the run when
overall line coverage is below the threshold and --min-area-coverage=N fails
it when a critical area (authentication and sessions, authorization, updates,
packages and extensions, migrations and install) drops below its threshold. The
CI coverage job installs PCOV and runs this automatically.
Database Matrix
The DatabaseMatrixTest.php tests compatibility across database engines. It refuses direct execution, so always run it through tests/run.php:
| Database | Status | How to Test |
|---|---|---|
| SQLite | ✓ Always tested | php tests/run.php DatabaseMatrix |
| MySQL 8.0 / 8.4 | ✓ When available | DB_DRIVER=mysql php tests/run.php DatabaseMatrix |
| MariaDB 10.6 / 10.11 / 11.4 | ✓ When available | Same as MySQL (driver auto-detects) |
Running Locally with MySQL
# Start MySQL
docker run -d --name mysql-test -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=bulletinbored_test -p 3306:3306 mysql:8.0
# Run tests
DB_DRIVER=mysql DB_HOST=127.0.0.1 DB_PORT=3306 DB_NAME=bulletinbored_test DB_USER=root DB_PASS=root php tests/run.php DatabaseMatrix
# Cleanup
docker stop mysql-test && docker rm mysql-test
Exit Codes
0— All tests passed1— One or more tests failed (or no test files found)
This makes the suite usable in CI/CD pipelines:
php tests/run.php && echo "OK" || echo "FAILED"
Adding New Tests
- Create
tests/YourFeatureTest.php require_once __DIR__ . '/harness.php';- Define test functions returning
Testobjects - Call
register_tests('test_foo', 'test_bar')at the bottom - Run with
php tests/run.php YourFeature