Firebase vs Forge: App Platform Developer Experience
For a better experience, this page will redirect to the interactive version of
this rubric. cancel
Criteria Overview
Introduction
This rubric provides criteria for assessing the development experience of an app platform. The intention is to form the basis of comprehensive and unbiased assessments of various app platforms in order to:
- Create a shared agreement and understanding of the health of various app platforms.
- Compare the health of app platforms with one another.
- Help the developers of app platforms determine the areas needing the greatest attention.
App Platform Types
For the purpose of this assessment framework, there are two types of app platforms:
- General Purpose App Platforms;- e.g. Firebase, PubNub or Heroku
- Product Extensibility App Platforms;- e.g. Jira Connect, Confluence Connect, Bitbucket Connect, Forge, Trello Power Ups or monday.com.
The assessment criteria relevant to Product Extensibility App Platforms includes all of the criteria of General Purpose App Platforms plus some extra criteria. All assessment criteria that are only applicable to Product Extensibility App Platforms are marked with "Specific to Product Extensibility App Platforms".
Viewing Assessments
To view assessments against the rubric, select one or more assessments from the drop down list titled Assessment visibility at the top left of this page. This will require others to have previously made assessments visible to you.
Comparing Assessments
Multiple assessments against the criteria can be view simultaneously. This allows separate platforms to be compared with one another and different assessments against a single platform to be compared. When viewing multiple assessments, they will appear side by side and an option will appear that allows for an interactive comparison through the use of a slider that controls a sensitivity setting which results in similar scores to be blurred thereby causing the differences to be highlighted.
Creating Assessments
To create assessments against the rubric, click the Create assessment button at the top right of this page. You can then choose who to share the assessment with.
Labels
The rubric includes a system of labels whereby each aspect is associated with one or more labels. This allows reports to be generated whereby the assessment data is grouped by label, thus providing further insights into the strengths and weaknesses of an app platform. The labels are as follows:
- Ease: Relates to how easy or difficult is it to develop apps using the app platform.
- Functionality: Relates to the functionality provided by the platform which can be utilized by apps.
- Trust: Relates to the level of customer trust in the apps using the app platform.
- Incentive: An app platform can incentivize developers in various ways such as financial gains, provision of growth expansion into related products or recognition.
Criteria Details
Area
Aspects
Documentation
Getting started guide
Show guidance- 0%
- No getting started guides exist.
- 33%
- A getting started guide exists, but does not lead through to more detailed guides other than the reference documentation.
- The information hierarchy is poor and confusing to navigate.
- 67%
- A getting started guide exists and leads through to one or two detailed guides.
- The getting started guide have some deep links into reference documentation.
- The information hierarchy is reasonably intuitive and simple to navigate.
- 100%
- One or more getting started guides are available, each tailored to a specific use case such as building a UI or product event driven app.
- The getting started guides deep links into reference documentation throughout.
- The getting started guides lead gently into other detailed guides and/or tutorials to avoid leaving the developer on a cliff at the end of the guide.
- The information hierarchy is intuitive and simple to navigate.
- The presentation of the guide makes it simple to copy code snippets (e.g. copy to buffer button).
- The documentation system may personalise code (e.g. using your access tokens and/or tenant details).
Reference documentation
Show guidance- 0%
- There is no reference documentation.
- 33%
- Reference documentation is often incomplete due to being manually produced.
- Reference documentation is only provided for REST APIs, but missing detail for other APIs such as event schemas, UI components and file formats.
- 67%
- Reference documentation is occasionally incomplete due to being manually produced.
- Reference documentation is provided for most types of APIs such as REST APIs, event schemas, UI components, file formats, modules, extension points, etc.
- The reference documentation supports deep linking.
- 100%
- The reference documentation is complete and accurate (e.g. generated from code).
- The reference documentation contains clear and detailed descriptions explaining all semantics.
- The reference documentation covers all types of APIs such as REST APIs, event schemas, UI components, file formats, modules, extension points, etc.
- The reference documentation supports deep linking.
- The reference documentation includes code snippets with example data.
- The presentation of the reference documentation makes it simple to copy code snippets (e.g. copy to buffer button).
- The reference documentation system may personalise code snippets (e.g. using your access tokens and/or tenant details).
Advanced guides and tutorials
Show guidance- 0%
- No reasonable advanced guides or tutorials exist.
- 33%
- An advanced guide or tutorial exists for some areas of functionality.
- The advanced guides and tutorials are poorly formatted such as in a single large page.
- 67%
- An advanced guide or tutorial exists for most major areas of functionality.
- Most advanced guides and tutorials are reasonably well formatted.
- 100%
- At least one advanced guide or tutorial exists for each major area of functionality.
- The advanced guides and tutorials are arranged into an intuitive information hierarchy.
- The format of all advanced guides and tutorials are consistent with one another.
Sample code
Show guidance- 0%
- Sample code does not exist for most major areas of functionality.
- Sample code is complex, convoluted or otherwise difficult to understand.
- 33%
- Sample code exists for a subset of functionality.
- Sample code may contain minor errors, preventing it from running.
- Sample code may be complex, convoluted or otherwise difficult to understand.
- 67%
- Sample code exists for most major areas of functionality.
- Sample code is clear and easy to understand.
- Sample code rarely contains errors and usually works against the latest APIs.
- 100%
- Sample code exists for all major areas of functionality.
- Sample code is clear and easy to understand.
- Sample code always works against the latest APIs.
- Sample code may be presented in executable form (e.g. story books).
- The presentation of the sample code makes it simple to copy code snippets (e.g. copy to buffer button).
- The documentation system may personalise code (e.g. using your access tokens and/or tenant details).
Changelog and announcements
Show guidanceRelease notes, aka changelogs, allows developers to keep up to date. They should contain deprecation notices as well as important updates and enhancements.
- 0%
- No release notes or announcements provided.
- Release notes and announcements are posted inconsistently (e.g. different locations, formats, etc) making them difficult to keep up with.
- 33%
- Brief release notes provided for only major or breaking changes.
- Announcements are provided intermittently, but sometimes features/changes land without appropriate notice.
- 67%
- Detailed release notes and announcements provided, but the presentation is basic.
- A basic non filterable release notes subscription mechanism may be provided such as RSS feed.
- 100%
- Detailed release notes and announcements provided.
- Release notes are presented such that they can be consumed very easily. e.g ordering by date and/or release ID, grouping by functional area, etc.
- An email based custom filterable release notes subscription mechanism is provided.
Development loop
App creation
Show guidance- 0%
- App creation is tedious and time consuming.
- Errors are typically encountered during app creation.
- 33%
- App creation is complex, requiring a number of steps to complete.
- Errors are often encountered during app creation.
- 67%
- App creation is simple, but may involve a number of steps.
- Errors are rarely encountered during app creation.
- Not iintuitive, some know-how is required.
- 100%
- App creation is simple and fast.
- Errors never occur during app creation.
- An intuitive, web based console provides app creation functionality.
App logging
Show guidance- 0%
- No logging.
- 33%
- Basic logging.
- Difficult to search and filter.
- 67%
- Advanced logging that supports searching and filtering.
- 100%
- Additionally very fast (real time) and/or dynamic filtering.
Debugging
Show guidance- 0%
- No debugging capability is provided. Developers must resort to logging.
- 33%
- Only a simple debugger can be used.
- Restrictions exist, preventing the debugging of certain situations.
- Developers often prefer logging due to restrictions.
- 67%
- Developers can use most standard tools providing debug capabilities such as an IDE or browser developer tools.
- Some restrictions may exist, preventing the debugging of certain situations.
- 100%
- Developers can easily use their favorite debugger, whether it be their IDE or browser developer tools.
- No debugging restrictions exist.
- Tooling is provided that enables certain situations to be easily injected to aid repeatability.
Tracing
Show guidance- 0%
- No tracing capabilities exist.
- 33%
- Simple tracing exists, but may not cover all functional areas.
- 67%
- Invocations can be traced throughout most the framework.
- Basic tooling exists to allow traces to be queried and viewed.
- 100%
- Invocations can be traced throughout the framework including spawned events.
- Intuitive tooling exists to allow traces to be queried and viewed.
- Tracing tooling provides textual and graphical means for presentation.
App management
Show guidance- 0%
- Only the developer of an app can manage it.
- 33%
- Apps can be managed by a team of developers.
- A distinct development team is required for each app which increases the amount of administration required.
- 67%
- Apps can be managed by a team of developers.
- A development team can be defined for managing multiple apps.
- 100%
- Apps can be managed by a team of developers.
- A development team can be defined for managing multiple apps.
- Members of an app management team can be assigned different roles such as to restrict important operations such as releases or viewing sensitive information.
App releases
Show guidance- 0%
- Deployment is manual, slow and opaque.
- 33%
- Deployment is simple and quick. Automated deployment is not supported.
- 67%
- Deployment is a breeze, but only supports a simple workflow.
- 100%
- Supports deployment to different environments such as development, testing, staging, production.
- App versions can be pre-released to trusted testers.
User interface features
UI capabilities
Show guidance- 0%
- The UI capabilities are severely restricted.
- Mobile is not supported.
- 33%
- The UI capabilities are adequate for the type app use cases.
- Mobile is support, but with significant limitations.
- Composability of UI components is limited or difficult to achieve.
- 67%
- Extensive client side UI capabilities are provided such that unanticipated app uses cases are typically covered.
- Apps render nicely in different platforms (e.g. web and mobile devices), but may separate code. e.g. Separate web and mobile libraries may be required as per the Firebase model.
- 100%
- All applicable client side capabilities can be utilised by the app.
- App render natively in web and mobile with minimal bespoke work for each platform.
Cross device UI compatibility and support
Show guidance- 0%
- Only one type of device (e.g. browser or iOS or Android) is supported.
- Only the most recent (one year) versions of the device is supported. For browser support, only the most popular browsers are supported.
- 33%
- Once device (e.g. browser) is well supported, but others are poorly supported.
- Good browser support, but basically no mobile support.
- 67%
- The UI can be developed for multiple devices, but there may be significant work to support each additional device.
- Possibly custom APIs or client client library for mobile devices to bridge gaps in capabilities.
- 100%
- The UI can be written once, but will render appropriately for each type of device.
- Alternatively, custom APIs and/or client libraries for mobile devices.
Capabilities
Extension points
Show guidanceThis consideration is specific to app platforms that extend a product. Mark this item as not applicable for assessments against app platforms that do not extend a product.
In this context, examples of extension points includes:
- Addition of panels to a product UI
- Addition of links/menu items to a product UI
- Addition of a field that can be displayed in a product UI
- Extending a product's search indexing
- Addition of the product’s data schema
- 0%
- Only one extension point is provided, severely limiting the use cases that can be implemented by apps.
- 33%
- 5 extension points are provided such as to satisfy use cases for a subset of functionality.
- 67%
- 10 extension points are provided. A significant proportion of the product's functionality can be extended.
- The location of extension points are well documented and/or discoverable.
- 100%
- 20 extension points are provided. Apps can extend the host product/platform in many ways.
- The location of extension points are simple to find (perhaps through documentation).
Persistence
Show guidance- 0%
- No persistence API is provided. App developers have to persist data using a third party service (if reachable), eroding trust due to user generated data being located in less known system.
- 33%
- A persistence mechanism/API is provided, but lacks basic features.
- Semantics are somewhat unclear and/or inconsistent, possibly leading to insecure practices in the developer community.
- The persistence API likely does not support transactions.
- 67%
- A persistence mechanism/API is provided which includes a comprehensive set of features.
- Semantics are reasonably clear, consistent and intuitive.
- Objects may be attached to parent entities to control their lifecycle.
- The persistence APIs are secure.
- The persistence API may not support transactions.
- The persistence API may only support simple queries.
- 100%
- One or more persistence APIs are provided each offering different strengths/semantics (e.g. relational, graph, etc).
- Libraries are provided to simplify calls to persistence APIs.
- Semantics are clear, consistent and intuitive.
- Objects may be attached to parent entities to control their lifecycle.
- The persistence APIs are secure by default, but can be relaxed to cater for various use cases.
- The persistence API supports transactions.
- The persistence API supports reasonably advanced queries covering most used cases.
- The persistence API support schema extension, including indexing and searchability.
Product and platform API invocation
Show guidance- 0%
- The framework does not allow apps to make API calls.
- 33%
- The framework allows apps to invoke a subset of product and platform APIs, possible due to limitations of scope and non support for cross product behavior.
- 67%
- The framework allows the app to invoke a range of product and platform APIs.
- The framework APIs are based on non-standard or dated technologies such as JWT authentication.
- 100%
- The framework allows the app to invoke all applicable product and platform APIs.
- The framework APIs are based on industry standards such as REST, OAuth 2.0, GraphQL, etc.
Webhooks
Show guidanceThis consideration is specific to app platforms that extend a product. Mark this item as not applicable for assessments against app platforms that do not extend a product.
- 0%
- Only a few webhook types supported. Difficult to debug. No filter support to prevent unnecessary calling of app.
- 33%
- Reasonable webhook types supported. OK dev loop. No filter support to prevent unnecessary calling of app.
- 67%
- Lots of webhook types supported. Easy dev loop. Possibly with filter support to prevent unnecessary calling of app.
- 100%
- All webhook types supported with a delightful dev loop. Filter support to prevent unnecessary calling of app.
Server side processing
Show guidance- 0%
- The framework does not support any form of background processing.
- 33%
- The framework allows for periodic background processing, but the functionality is not built in. For example, an external cron like service is needed for the triggers.
- The framework does not support long running tasks. For example, invocation timeouts can make it very difficult to reliably implement long running jobs.
- 67%
- The framework has built in support for periodic background processing. The framework provides an API for defining and dynamically changing the periodic processing.
- The framework does not support long running tasks. For example, invocation timeouts can make it very difficult to reliably implement long running jobs.
- 100%
- The framework has built in support for periodic background processing. The framework provides an API for defining and dynamically changing the periodic processing.
- The framework provides direct support for apps to perform long running jobs. For example, APIs are provided allowing jobs to be scheduled, queried and used as the input for other operations/triggers.
Non functional features
Access and egress of User Generated Data (UGC) and Personal Data (PD). (specific to Product Extensibility App Platforms)
Show guidance- 0%
- Any installed app can access and egress of UGC and PD.
- The framework does not have control nor visibility into the egress of UGC and PD.
- It's is possible to leak privileged content through permission elevation (app exposes privileged data to a non authorised user).
- 33%
- Only apps with particular capabilities and scopes can access UGC and PD.
- The framework has some visibility into the egress of UCG and/or PD.
- It is possible to understand where permission elevation occurs, but the information is not forthcoming.
- 67%
- Only apps with particular capabilities and scopes can access UGC and PD.
- The framework provides complete control and visibility into the access and egress of UGC and PD.
- The app permission model is course resulting in a lack of clarity over specific permissions required by an app.
- The framework provides app developers and administrators insights or warnings into permission elevation.
- 100%
- The framework provides complete control and visibility into the access and egress of UGC and PD.
- The framework gracefully reduces the capability to install certain apps based on the app's trust profile and the role of the user attempting to install it.
- The framework ensures the person installing an app is aware of all permissions the app is requesting.
- It's is impossible to leak privileged content through permission elevation (app exposes privileged data to a non authorised user).
Reliability
Show guidance- 0%
- APIs break occasionally without warning.
- 33%
- Framework features have a P99 availability.
- Non backward compatible changes to APIs are accompanied with minimal notice and migration guides.
- 67%
- Framework features have a P99.5 availability.
- Non backward compatible changes to APIs are planned with reasonable notice and migration guides.
- 100%
- Framework features have a P99.9 or better availability.
- Non backward compatible changes to APIs are always planned well. Deprecation notices and clear migration guides are provided.
Performance
Show guidance- 0%
- Known performance issues that apps can not work around.
- 33%
- App developers have to employ advanced techniques tp get performance out of their apps.
- 67%
- APIs allow apps to be fast, but may require extra work by app developers to take advantage of.
- 100%
- APIs naturally encourage apps to be fast through techniques such as injecting context into invocations and allowing caching.
Industry standards
Show guidance- 0%
- APIs exposed by the app platform do not conform to industry standard protocols and are difficult to use/understand.
- 33%
- APIs exposed by the app platform are a mix of industry standards and non-standard protocols.
- 67%
- Most APIs exposed by the app platform are based on industry standard protocols (e.g. OAuth 2.0).
- The app platform allows for, but does not encourage the appropriate use of industry standard protocols such as client authorization flows for interactions on behalf of users.
- 100%
- All APIs exposed by the app platform are based on industry standard protocols (e.g. OAuth 2.0).
- The app platform encourages the appropriate use of industry standard protocols such as client authorization flows for interactions on behalf of users.
Administration and monitoring
App installation
Show guidanceThis consideration is specific to app platforms that extend a product. Mark this item as not applicable for assessments against app platforms that do not extend a product.
- 0%
- App installation involves several steps.
- Only administrators can install apps.
- 33%
- App installation takes only one or two quick steps.
- Only administrators can install apps.
- 67%
- App installation is quick, involving only a single step.
- It may be possible for administrators to enable the ability for non-administrators to install apps.
- 100%
- There are multiple entry points into installing apps such as app suggestions based on context within a product or workflow.
- App installation is quick, involving only a single step.
- It is possible for administrators to enable the ability for non-administrators to install apps.
Error monitoring
Show guidance- 0%
- No error monitoring capabilities are provided.
- 33%
- Minimal error monitoring.
- 67%
- Two types of basic error monitoring such as crash reports and new issue alerts.
- 100%
- Automatic provision of actionable insight into app issues.
- Curated crash reports: synthesis of crashes into a manageable list of issues with contextual information and highlighting of the severity and prevalence of errors.
- Realtime alerts for new issues, regressed issues, and growing issues that might require immediate attention.
- Integration with app analytics.
Performance monitoring
Show guidance- 0%
- None.
- 33%
- HTTP network requests.
- 67%
- Two types of basic performance monitoring such as HTTP network requests and time to paint.
- 100%
- HTTP network requests.
- Time to paint.
- Performance metrics broken down by attributes, like country, device, app version, and OS level.
- Customized monitoring to capture the app's performance in specific situations.
Administrator Monitoring
Show guidanceThis consideration is specific to app platforms that extend a product. Mark this item as not applicable for assessments against app platforms that do not extend a product.
- 0%
- Administrators have no insight into app behavior (APIs being used, performance, reliability, etc).
- 33%
- Administrators have basic insight into app behavior (APIs being used, performance, reliability, etc).
- 67%
- Administrators have reasonable insight into app behavior (APIs being used, performance, reliability, etc).
- Administrators may be able to block some APIs from apps.
- 100%
- Administrators have complete insight into app behavior (APIs being used, performance, reliability, etc).
- Administrators may be able to block some APIs from apps.
App analytics
Show guidance- 0%
- The app platform does not provide any app analytics to app developers.
- 33%
- The app platform provides basic app analytics to app developers, consisting of 1 to 3 primary views. The analytics may lack app platform context.
- The presentation of app analytics includes basic charts and tables of data.
- There is no ability to drill down or filter data.
- 67%
- The app platform provides a variety of app analytics to app developers, consisting of 4 to 10 views which includes context from the app platform where appropriate. Example app platform context may include product context such as extension point location, invocation source context and license type.
- The presentation of app analytics includes a variety of formats such as charts and maps showing geo based data.
- For most views, there is no ability to drill down or filter data.
- 100%
- The app platform provides comprehensive app analytics to app developers which includes context from the app platform where appropriate. Example app platform context may include product context such as extension point location, invocation source context and license type.
- The presentation of app analytics includes a variety of graphical formats such as charts and maps showing geo based data.
- The presentation of app analytics allows for drilling down and filtering.
Status
Show guidanceThis consideration relates to the provision of status information relating to the app platform.
- 0%
- No status information is provided.
- 33%
- A status information dashboard is provided, but it lacks detail and is sometimes not up to date.
- The status information is regularly over an hour out of date.
- 67%
- A status information dashboard is provided.
- The status information is reflects the current state within an hour.
- The categories in which status is reported may not be intuitive.
- 100%
- A detailed status information dashboard is provided and kept up to date.
- The status information is reflects the current state within 5 minutes.
- The categories in which status is reported are intuitive.
Commerce
Running costs
Show guidance- 0%
- The platform "charges like a wounded bull".
- 33%
- The platform provides a free tier for a limited period of time (one year or less).
- The platform charges above the free tier are significant for "side projects" and very small organisations (minimum of $500 USD p.a. for the smallest resource consumption possible).
- 67%
- The platform provides a free tier sufficient to try all features on an ongoing basis.
- Platform charges above the free tier are linear and will not result in "bill shock".
- 100%
- The platform is free.
Marketing and distribution
Show guidance- 0%
- No built in support for marketing and distributing apps.
- 33%
- Minimal support for marketing and distributing apps.
- 67%
- Good support for marketing and distributing apps.
- 100%
- Superb support for marketing and distributing apps.
Pricing models
Show guidanceThis consideration is specific to app platforms that extend a product. Mark this item as not applicable for assessments against app platforms that do not extend a product.
- 0%
- No pricing models supported.
- 33%
- Basic, single pricing model.
- 67%
- Multiple pricing models such as per tenant, per user, resource usage, etc.
- 100%
- Multiple pricing models that can be used in conjunction.
Payment models
Show guidanceThis consideration is specific to app platforms that extend a product. Mark this item as not applicable for assessments against app platforms that do not extend a product.
- 0%
- Very weak APIs relating to payment.
- 33%
- Good payment APIs, but with a single payment model.
- 67%
- Solid payment APIs/platform with two simple models.
- 100%
- Great payment APIs/platform with multiple models.
Support
Community support
Show guidance- 0%
- No support exists.
- 33%
- Somewhat active and helpful community.
- Barely any participation by company representatives.
- 67%
- Quite active and helpful community.
- Some ad-hoc participation by company representatives such as product managers, engineers and developer advocates.
- 100%
- Very active and helpful community.
- Significant participation by company representatives such as product managers, engineers and developer advocates.
Official support
Show guidance- 0%
- Ad-hoc support without SLAs.
- Developers are unable to create nor see issues.
- 33%
- Part time, best effort support without SLAs.
- Developers are able to create issues, but have no visibility into their status.
- 67%
- Dedicated support team without SLAs.
- Developers are able to create issues, but may have limited visibility into their status.
- 100%
- Dedicated support team with SLAs.
- Developers are able to create and see issues to track their status.
In person events
Show guidance- 0%
- No in-person events exist for app developers.
- 33%
- Ad-hoc in-person events occur.
- The events are by invitation only and most likely affordable by commercially successful app developers.
- In-person events have a poor ratio of experts to app developers.
- 67%
- Regular in-person events exist, but may be by invitation only or may be prohibitively expensive to attend by some app developers depending on their revenue and geographic location.
- In-person events have a decent ratio of experts to app developers.
- 100%
- Regular in-person events are available for attendance by any app developer.
- In-person events have a high ratio of experts to app developers.