›
This is the forum archive of Homey. For more information about Homey, visit the Official Homey website.

The Homey Community has been moved to https://community.athom.com.

This forum is now read-only for archive purposes.

API.AI, Google's Machine Learning Platform

MathijsFrenckenMathijsFrencken Member
edited January 2017 in Ideas & Suggestions
Hey guys,
I have just hooked Homey up to the Google service: API.AI ( http://api.ai ), to make homey a bit smarter.

API.ai works with intents (questions) and entities (words it knows) and machine learning. This enabled you to let homey 'learn commands' that are related to or contain keywords of other commands. Therefore making homey smarter with every speech command.

Visit API.ai to see what else is possible (like small talk support and one-click integrations with other popular services.)

Now my question: Do you think the community is interested in learning how this is done or will I just throw 2 hours away by showing you?

Comments

  • MHubertMHubert Member
    maybe you could post a video which shows the end results and when people are hysterical about the made progress?
    And then its a good point to explain how you did it ;)
  • AaronAaron Member
    Sounds good but I would also like to see it to understand it better.

    thanks
    Aaron
  • mruitermruiter Member
    Beter yet , share the code and the manual :) . But indeed, love to see it in action.

  • bodyartbodyart Member
    Seems to become very interesting.
  • MathijsFrenckenMathijsFrencken Member
    Just a quick example:


    This image shows for example how I might give the command to stream my favorite TV show (family guy). The colored words let Homey know that there are synonyms availible for these commands and there are multiple and not exactly specified ways to give the command.
  • ArenBreurArenBreur Member
    I am interested, a video with results would be very nice

    thanks in advance
  • Denn1992Denn1992 Member
    Would love to implement Homey with this API. Everything to increase the voice recognition is welcome :smile: 

    Thanks!
  • JustinVanElstJustinVanElst Member
    Interested!
  • IcarusIcarus Member
    I'm very much interested in how you connected both services. A quick HOWTO maybe?
  • MathijsFrenckenMathijsFrencken Member
    I will try to make a short tutorial or video in the upcoming weekend, right now im gonna leave the programmers with this: https://www.npmjs.com/package/apiai  if you want to maybe try to connect it yourself! (not that difficult, there is a module that only requires two tokens from your dashboard.)
  • FloodnlFloodnl Member
    @MathijsFrencken thanks a lot for your efforts in advanced,  i really like this idea.  Do notice that some users (like me)  are clueless about coding so make a (jip en janneke) manual please :wink:
  • JustinVanElstJustinVanElst Member
    edited January 2017
    I dont think we wil be able to use this to enhance the voice command recognition? I was looking into a Alexa app. But i dont believe we have direct access to the microphone?
    https://github.com/alexa/alexa-skills-kit-sdk-for-nodejs
    https://github.com/alexa/alexa-avs-sample-app
  • ZperXZperX Member
    edited January 2017
    APIAI is not a speech to text engine. You don`t need the microphone. But I it would require a good speech recognition from Homey and unfortunately that is not on the horizon.
  • MathijsFrenckenMathijsFrencken Member
    Icarus said:
    I'm very much interested in how you connected both services. A quick HOWTO maybe?
    A quick howto is possible:

    -- Quick disclaimer: I'm not a programmer, just interested in Homey and machine learning. Don't question my approach, I know many of you can do it WAY better and I urge you to try :) --


    How the app I made kind of works 'Houwtje touwtje'
    I made an app, this app fires when homey gets a string 'lights off apiai', but no-one wants to say their 'command + apiai'.

    So I made a flow that duplicates every command and adds 'apiai' after the string. (only if it doesn't contain 'apiai' part yet!!) (infinite loop warning haha)
     
    'Turn off the lights now' becomes 'turn off the lights now apiai'

    The app fires (triggered by the word 'apiai'), and it includes the module availible here: https://www.npmjs.com/package/apiai    (or https://github.com/api-ai/api-ai-javascript for web-applications )

    Then it extracts the command (minus the 'apiai') and sends it to the api.ai service.


    How the API.ai service tries to be smart about it
    The apiai service checks if an intent for this command exists (or for the structure of this sentence by checking known synonyms) and returns: 

    - One of many specified responses.
    - An action to be taken (can be handled by the app and then by Homey because api.ai itself can't execute this action)
    - It can even ask follow up questions if for example a parameter that you checked in APIAI.ai as 'required' for example:

    - "stream family guy"
    - "Where you you want to stream this to?"
    - "The living room please".

    A programmer now has two json responses(returned to the written app), one with the action and a follow up, answered question.

    How the app almost instantly gets a JSON response that programmers can use for so much different stuff
    The JSON response can look something like this:
    {
      "id": "cf74c595-6a7b-401e-890b-9f0a23ce7b57",
      "timestamp": "2017-01-10T14:41:15.359Z",
      "result": {
        "source": "agent",
        "resolvedQuery": "Stream Family guy naar woonkamer",
        "action": "familyguy",
        "actionIncomplete": false,
        "parameters": {
          "Apparaten": "tv",
          "Kamer": "woonkamer",
          "Kan": "",
          "*******": "family guy",
          "*******": "",
          "TV": ""
        },
        "contexts": [],
        "metadata": {
          "intentId": "ab39c2f4-7c85-46e9-9890-d0661d5e5b7a",
          "webhookUsed": "false",
          "webhookForSlotFillingUsed": "false",
          "intentName": "Family Guy"
        },
        "fulfillment": {
          "speech": "Oke ik zet family guy op.",
          "messages": [
            {
              "type": 0,
              "speech": "Oke ik zet family guy op."
            }
          ]
        },
        "score": 1
      },
      "status": {
        "code": 200,
        "errorType": "success"
      },
      "sessionId": "26765679-34a0-43f1-bbb4-11529fcf4834"
    }

    See that (currently empty) context part? That is where you can set a context for the dialog that you are having with homey.

    What the downsides of using this are:
    - Built in commands now generate TWO responses.(one from homey, one from API.ai) A workaround for this is make an intent with ALL built in commands and simply leave the response for these commands EMPTY.
    - Dutch domains are currently still in development, when using english you can just 'flip on' domains that intelligently deal with A LOT of pre-defined queries by api.ai. English is easier at this moment.
    - Reading the docs on how to configure and get started can be overwhelming at first.

    The UPSIDES of this!
    - Homey gets SMARTER as you give commands! The adding of entities (synonyms) make it recognize a wider range of commands and select the right response + action!
    - You can view a log of what was decided and see where improvements can be made.
    - When Dutch domains (pre-built intelligence) are released, Homey will get so much smarter instantly, I've tried the english variant, it's amazing.
    - DIALOG with HOMEY! Need I say more?
    - Integration with Skype, facebook messenger, telegram and other services requires only one click.
    - Webhooks with existing services.

    Watch THIS video for a 3 minute video explaining the intelligence of the platform.

    This turned out to be longer then expected, oops!


    The possibilities are endless but I have exams coming up so I don't have the time to develop this further at this time.

    My apologies in advance for my (probably) terrible english.

    If you have any questions, I will be in this thread.

    Thank you.

  • MathijsFrenckenMathijsFrencken Member
    Icarus said:
    I'm very much interested in how you connected both services. A quick HOWTO maybe?
    See my previous postFloodnl said:
    @MathijsFrencken thanks a lot for your efforts in advanced,  i really like this idea.  Do notice that some users (like me)  are clueless about coding so make a (jip en janneke) manual please :wink:
    I'm gonna try to develop an app but im also not that skilled in programming :)

    JustinVanElst said:
    I dont think we wil be able to use this to enhance the voice command recognition? I was looking into a Alexa app. But i dont believe we have direct access to the microphone?
    https://github.com/alexa/alexa-skills-kit-sdk-for-nodejs
    https://github.com/alexa/alexa-avs-sample-app
    It might be possible to enhance speech recognition but I think that requires Athom to look into this.

    ZperX said:
    APIAI is not a speech to text engine. You don`t need the microphone. But I it would require a good speech recognition from Homey and unfortunately that is not on the horizon.
    True but even at failed Homey-commands: API.AI can be trained to make something out of what it heard if it is only a little related to what you meant when you gave the command. The more information you input the smarter it (should) get.
  • FloodnlFloodnl Member
    @Emile is this something that you can use improving the homey? 
  • DaneeDeKruyffDaneeDeKruyff Member

    Kuddos to @mathijsFrencken !

  • KoenMartensKoenMartens Member
    @MathijsFrencken Great app idea. Please take into account that the standard (free) 'subscription' of api.ai is capabable of ~1 request per second (https://api.ai/pricing/). This will be a severe hinder at times if multiple users will be using it.
    I think if everyone fills in their 'personal' key (or whatever api.ai has you enter), you cannot transfer the api.ai-part of this app, right? The intents and whatever.
  • MathijsFrenckenMathijsFrencken Member
    KoenMartens said:
    @MathijsFrencken Great app idea. Please take into account that the standard (free) 'subscription' of api.ai is capabable of ~1 request per second (https://api.ai/pricing/). This will be a severe hinder at times if multiple users will be using it.
    I think if everyone fills in their 'personal' key (or whatever api.ai has you enter), you cannot transfer the api.ai-part of this app, right? The intents and whatever.


    There is an export/import function, I don't know it's limitations though!

    And Homey isnt't fast enough to process more then 1 request per second :)
  • jeroenbos22jeroenbos22 Member
    Would be Nice to add the wisdom domain. Unfortunatly only in english but we could make a Trigger for the word question So you can do:  Ok homey, question. What is the height of the Eifel tower? 

    That Homey can awnser thousands of questions out of the box. See https://docs.api.ai/docs/encyclopedia-knowledge-wisdom
  • EvertorNEvertorN Member
    For Google users it's fairly easy to improve speech recognition. If you have a Pixel or Google Home, you can use the Google Assistant + IFTTT to use that for speech recognition:

    - Integrate IFTTT for Google with keyword "Tell Homey to <variable>"
    - Send <variable> to Homey using IFTTT
    - Create a Flow that gets triggered by IFTTT and uses the Microphone action 'Emulate speech' and pass the <variable>
  • BumblezBumblez Member
    Kind of sad that we have to resort to workarounds like these to get Homey to recognize our spoken sentences.

    And that even though the website still claims: "These two state-of-the-art microphones enable Homey to listen to your voice, crisp and clear." (or, in Dutch: "Met deze twee state-of-the-art microfoons kan Homey je stem altijd kraakhelder horen.")

Sign In or Register to comment.