Last month, we presented our work Speculate at the ACM Foundations of Software Engineering (FSE) conference, held at Montreal, Canada. Speculate extracts OpenAPI specifications from a webserver source code.
Modern servers can be written in a variety of languages and frameworks, such as Django or FastAPI for Python, and Jersey or SpringBoot for Java. For example, the following Django webserver implements one endpoint, GET:/movies/, taking two query parameters year and limit, and returning either a list of movies or an error.
1 │ # urls.py
2 │ urlpatterns = [path("movies/", movie_list)]
3 │
4 │ # views.py
5 │ def movie_list(request):
6 │ if "year" not in request.GET: # 1. 'required' is an if
7 │ return JsonResponse({"error": "year is required"}, status=400)
8 │ year = int(request.GET["year"]) # 2. 'type: integer' is a conversion
9 │
10 │ limit = int(request.GET.get("limit", 10)) # 3. 'default: 10' is a fallback
11 │ if limit > 100: # 4. 'maximum: 100' is an if
12 │ return JsonResponse({"error": "limit too large"}, status=400)
13 │
14 │ movies = Movie.objects.filter(year__gte=year)[:limit]
15 │ return JsonResponse( # 5. the response schema
16 │ [{"title": m.title, "year": m.year} for m in movies],
17 │ safe=False)The OpenAPI specification is the server’s contract and describes the various REST endpoints, their input and output schemas, and the constraints on them. Here is the OpenAPI specification of the above webserver:
1 │ /movies/: # ── the URL
2 │ get: # ── the HTTP method
3 │ parameters: # ── the input parameters
4 │ - name: year
5 │ in: query
6 │ required: true # ── constraint: year must be present
7 │ schema:
8 │ type: integer # ── constraint: year must be an integer
9 │
10 │ - name: limit
11 │ in: query
12 │ required: false
13 │ schema:
14 │ type: integer
15 │ default: 10 # ── constraint: limit defaults to 10
16 │ maximum: 100 # ── constraint: limit can maximum be 100
17 │
18 │ responses: # ── the outputs
19 │ '200': # ── success: a list of movies
20 │ content:
21 │ application/json:
22 │ schema:
23 │ type: array
24 │ items:
25 │ type: object
26 │ properties:
27 │ title: {type: string}
28 │ year: {type: integer}
29 │ '400': # ── failed query
30 │ description: Invalid parametersOpenAPI specifications power an entire ecosystem of tooling such as auto-generated client libraries [4], test suites [5, 6], and mock servers [7]. However, keeping specification in-sync with the webserver code is tedious. Therefore, it is often found that manually-written OpenAPI specifications become obsolete and even incorrect, as webserver code evolves.
Existing tools that automatically generate the specifications from code fall under two categories: dynamic analysis and static analysis.
Dynamic analysis tools, such as AppMap and ApiCarv, let the developer exercise various APIs while the tools capture API interactions to generate observed endpoints, parameters, and responses. However, these tools often fall short on the coverage of the specifications: it only covers endpoints that are actually exercised by the developer. They fail to capture all the parameters constraints and optional parameters. For example in the webserver above, if the developer never exercises the GET/movies API with limit>100, these tools may not add the parameter constraint that the maximum allowed value for limit is 100.
Static analysis tools are usually deeply coupled with the language and framework, for example drf-spectacular for Django-DRF and springdoc-openapi for SpringBoot. However, correctly generating these specifications from code is challenging. For example in the above webserver, a static analysis tool will need to infer required: true for the year parameter, from if "year" not in request.GET: condition. For production webservers, these inferences may need to happen across all program paths, which may be split across multiple files and function calls.
The current state-of-the-art static analysis tool is Respector [1], published at ICSE 2024. In brief, Respector walks every program path, collecting the conditions along the way, and classifies each path as “ends in 200” or “ends in 400” status codes. For example, the above webserver has three paths: missing year → 400 (line 7), limit > 100 → 400 (line 12), and everything else → 200. Respector then invokes Z3 theorem prover to combine conditions across all the successful paths, which it then translates into an OpenAPI specification: year is required: true, and limit has maximum: 100. See [1] for more details.
While highly sophisticated, Respector suffers with the issues that plague all the static analysis approaches:
- Dynamic behaviors and knowledge boundaries: Dynamic behaviours like interfaces resolved by injection, reflection, type-erased generics, library code outside the server repository, and foreign function calls create knowledge boundaries for Respector. For example in enviroCar, the developer writes only an interface,
PaginationProvider, and Guice, a dependency injection framework, plugs in the real class,PaginationProviderImplat startup. In such cases, the Respector’s OpenAPI specifications degrade. - Path explosion: While our toy example has only three program paths, Senzing’s /entity-networks API implementation has millions of possible paths. This path explosion led Respector towards an incorrect specification.
- Tedious Maintenance: Z3 cannot represent string operations like splitting on commas, leaving 58 constraints in Ohsome repository impossible to state. Similarly, Respector does not describe response constraints. Fixing Z3’s limitations and extending Respector require significant expertise. Finally, Respector is deeply coupled with Java. It is tedious to extend it to other languages and frameworks, such as Django for Python.
We build and evaluate Speculate [2] with a much simpler design than Respector. Yet, Speculate outperforms Respector on all dimensions for production Spring boot and Jersey repositories. The recalls for request and response constraints jump from ~10% for Respector to ~80% for Speculate.
Speculate performs lightweight static analysis to index the webserver code and then sends relevant code to an LLM for generating the specification. We demonstrate that Speculate does not suffer from the same challenges as Respector:
- Speculate is easy to generalize across languages and frameworks as it requires only lightweight static analysis; Speculate currently supports Python’s Django, and Java’s Jersey and Spring Boot.
- Speculate surpasses knowledge boundary issues: LLMs may have seen unavailable library code or foreign function calls in its training, and can also infer reflection and other dynamic behavior.
- Speculate is easy to extend: while adding response constraints is highly challenging in Respector as it requires adding more static analysis, it was straightforward to do in Speculate.
Overall, we found it interesting that while there are well-founded fears around LLMs making software maintenance tasks difficult for the programmers due to their non-determinism and hallucinations; Speculate provides an opposing data point. LLMs can help us create much simpler, easier-to-extend tools, while outperforming existing tools. Such LLM-based tools might provide free lunch: unlike Respector, Speculate’s accuracy might automatically improve as LLMs evolve in their reasoning capability.
Speculate is available at [3]. Please feel free to give it a try and in case of any questions reach out to us: Krishanu (krishanu.visitor@iitd.ac.in) and Kushagra (csy247554@iitd.ac.in).