Added test to enforce that all supported annotations are documented (#4723)

* Added test to enforce that all supported annotations are documented

Well, at least mentioned.

Refs vimeo/psalm#3816

* Type things

* Make things pretty

* Only check @psalm- annotations, group

* Add documentation for `@psalm-require-extends` and `@psalm-require-implements`

* Dropped logicalOr that has become redundant

* Add explicit tag

* Document @psalm-template

* Add @psalm-template-covariant

* Document `@psalm-method`

* Add list of undocumented docblock annotations

Co-authored-by: Matthew Brown <github@muglug.com>

docs/annotating_code/supported_annotations.md

@@ -7,7 +7,7 @@ Psalm supports a wide range of docblock annotations.
 Psalm uses the following PHPDoc tags to understand your code:
 
 - [`@var`](https://docs.phpdoc.org/latest/references/phpdoc/tags/var.html)
-  Used for specifying the types of properties and variables
+  Used for specifying the types of properties and variables@
 - [`@return`](https://docs.phpdoc.org/latest/references/phpdoc/tags/return.html)
   Used for specifying the return types of functions, methods and closures
 - [`@param`](https://docs.phpdoc.org/latest/references/phpdoc/tags/param.html)
@@ -68,7 +68,7 @@ function addFoo(?string &$s) : void {
 }
 ```
 
-### `@psalm-var`, `@psalm-param`, `@psalm-return`, `@psalm-property`, `@psalm-property-read`, `@psalm-property-write`
+### `@psalm-var`, `@psalm-param`, `@psalm-return`, `@psalm-property`, `@psalm-property-read`, `@psalm-property-write`, `@psalm-method`
 
 When specifying types in a format not supported by phpDocumentor ([but supported by Psalm](#type-syntax)) you may wish to prepend `@psalm-` to the PHPDoc tag, so as to avoid confusing your IDE. If a `@psalm`-prefixed tag is given, Psalm will use it in place of its non-prefixed counterpart.
 
@@ -475,6 +475,38 @@ class User {
 }
 ```
 
+### `@psalm-require-extends`
+
+The @psalm-require-extends-annotation allows you to define a requirements that a trait imposes on the using class.
+
+```php
+abstract class DatabaseModel {
+  // methods, properties, etc.
+}
+
+/**
+ * @psalm-require-extends DatabaseModel
+ */
+trait SoftDeletingTrait {
+  // useful but scoped functionality, that depends on methods/properties from DatabaseModel
+}
+
+
+class MyModel extends DatabaseModel {
+  // valid
+  use SoftDeletingTrait;
+}
+
+class NormalClass {
+  // triggers an error
+  use SoftDeletingTrait;
+}
+```
+
+### `@psalm-require-implements`
+
+Behaves the same way as `@psalm-require-extends`, but for interfaces.
+
 ## Type Syntax
 
 Psalm supports PHPDoc’s [type syntax](https://docs.phpdoc.org/latest/guides/types.html), and also the [proposed PHPDoc PSR type syntax](https://github.com/php-fig/fig-standards/blob/master/proposed/phpdoc.md#appendix-a-types).

docs/annotating_code/templated_annotations.md

@@ -68,9 +68,9 @@ class One_off_instance_of_MyContainer {
 
 This pattern can be used in large number of different situations like mocking, collections, iterators and loading arbitrary objects. Psalm has a large number of annotations to make it easy to use templated types in your codebase.
 
-## `@template`
+## `@template`, `@psalm-template`
 
-The `@template` tag allows classes and functions to declare a generic type parameter.
+The `@template`/`@psalm-template` tag allows classes and functions to declare a generic type parameter.
 
 As a very simple example, this function returns whatever is passed in:
 
@@ -380,7 +380,7 @@ function takesDogList(Collection $dog_collection) : void {
 
 Here we're not doing anything bad – we're just iterating over an array of objects. But Psalm still gives that same basic error – "getNoises expects a `Collection<Animal>`, but `Collection<Dog>` was passed".
 
-We can tell Psalm that it's safe to pass subtypes for the templated param `T` by using the annotation `@template-covariant T`:
+We can tell Psalm that it's safe to pass subtypes for the templated param `T` by using the annotation `@template-covariant T` (or `@psalm-template-covariant T`):
 
 ```php
 <?php

src/Psalm/DocComment.php

@@ -22,7 +22,7 @@ use function strspn;
 
 class DocComment
 {
-    private const PSALM_ANNOTATIONS = [
+    public const PSALM_ANNOTATIONS = [
         'return', 'param', 'template', 'var', 'type',
         'template-covariant', 'property', 'property-read', 'property-write', 'method',
         'assert', 'assert-if-true', 'assert-if-false', 'suppress',

tests/DocumentationTest.php

@@ -1,39 +1,84 @@
 <?php
+
 namespace Psalm\Tests;
 
+use DOMAttr;
+use DOMDocument;
+use DOMXPath;
+use PHPUnit\Framework\Constraint\Constraint;
+use Psalm\Config;
+use Psalm\Context;
+use Psalm\DocComment;
+use Psalm\Internal\RuntimeCaches;
+use Psalm\Tests\Internal\Provider;
+
+use function array_filter;
 use function array_keys;
+use function array_shift;
 use function count;
-use const DIRECTORY_SEPARATOR;
-use const LIBXML_NONET;
 use function dirname;
 use function explode;
 use function file_exists;
 use function file_get_contents;
+use function glob;
 use function implode;
+use function in_array;
 use function preg_quote;
-use Psalm\Config;
-use Psalm\Context;
-use Psalm\Tests\Internal\Provider;
 use function sort;
 use function strpos;
+use function str_replace;
 use function substr;
 use function trim;
-use function glob;
-use function str_replace;
-use function array_shift;
-use DOMDocument;
-use DOMXPath;
-use DOMAttr;
-use Psalm\Internal\RuntimeCaches;
-
-use function array_filter;
 use function var_export;
 
+use const DIRECTORY_SEPARATOR;
+use const LIBXML_NONET;
+
 class DocumentationTest extends TestCase
 {
+    /**
+     * a list of all files containing annotation documentation
+     */
+    private const ANNOTATION_DOCS = [
+        'docs/annotating_code/supported_annotations.md',
+        'docs/annotating_code/templated_annotations.md',
+        'docs/annotating_code/adding_assertions.md',
+        'docs/security_analysis/annotations.md',
+    ];
+
+    /**
+     * annotations that we don’t want documented
+     */
+    private const INTENTIONALLY_UNDOCUMENTED_ANNOTATIONS = [
+        '@psalm-self-out', // I'm fairly sure it's intentionally undocumented, but can't find the reference
+        '@psalm-variadic',
+    ];
+    
+    /**
+     * These should be documented
+     */
+    private const WALL_OF_SHAME = [
+        '@psalm-assert-untainted',
+        '@psalm-consistent-constructor',
+        '@psalm-flow',
+        '@psalm-generator-return',
+        '@psalm-ignore-variable-method',
+        '@psalm-ignore-variable-property',
+        '@psalm-override-method-visibility',
+        '@psalm-override-property-visibility',
+        '@psalm-scope-this',
+        '@psalm-seal-methods',
+        '@psalm-stub-override',
+        '@psalm-taint-unescape',
+        '@psalm-yield',
+    ];
+
     /** @var \Psalm\Internal\Analyzer\ProjectAnalyzer */
     protected $project_analyzer;
 
+    /** @var string */
+    private static $docContents = '';
+
     /**
      * @return array<string, array<int, string>>
      */
@@ -295,4 +340,68 @@ class DocumentationTest extends TestCase
             "Duplicate shortcodes found: \n" . var_export($duplicate_shortcodes, true)
         );
     }
+
+    /** @dataProvider knownAnnotations */
+    public function testAllAnnotationsAreDocumented(string $annotation): void
+    {
+        if ('' === self::$docContents) {
+            foreach (self::ANNOTATION_DOCS as $file) {
+                self::$docContents .= file_get_contents(__DIR__ . '/../' . $file);
+            }
+        }
+
+        $this->assertThat(
+            self::$docContents,
+            $this->conciseExpected($this->stringContains('@psalm-' . $annotation)),
+            "'@psalm-$annotation' is not present in the docs"
+        );
+    }
+
+    /** @return iterable<string, array{string}> */
+    public function knownAnnotations(): iterable
+    {
+        foreach (DocComment::PSALM_ANNOTATIONS as $annotation) {
+            if (in_array('@psalm-' . $annotation, self::INTENTIONALLY_UNDOCUMENTED_ANNOTATIONS, true)) {
+                continue;
+            }
+
+            if (in_array('@psalm-' . $annotation, self::WALL_OF_SHAME, true)) {
+                continue;
+            }
+
+            yield $annotation => [$annotation];
+        }
+    }
+
+    /**
+     * Creates a constraint wrapper that displays the expected value in a concise form
+     */
+    public function conciseExpected(Constraint $inner): Constraint
+    {
+        return new class ($inner) extends Constraint
+        {
+            /** @var Constraint */
+            private $inner;
+
+            public function __construct(Constraint $inner)
+            {
+                $this->inner = $inner;
+            }
+
+            public function toString(): string
+            {
+                return $this->inner->toString();
+            }
+
+            protected function matches($other): bool
+            {
+                return $this->inner->matches($other);
+            }
+
+            protected function failureDescription($other): string
+            {
+                return $this->exporter()->shortenedExport($other) . ' ' . $this->toString();
+            }
+        };
+    }
 }